[Paper Review] Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks

Nicolas Usunier, Gabriel Synnaeve+2 more|arXiv (Cornell University)|Sep 10, 2016

Artificial Intelligence in Games8 references102 citations

TL;DR

The paper introduces StarCraft micromanagement benchmarks for RL and a heuristic episodic exploration method that randomizes policy parameters at episode start, enabling learning with deterministic policies where Q-learning and REINFORCE struggle.

ABSTRACT

We consider scenarios from the real-time strategy game StarCraft as new benchmarks for reinforcement learning algorithms. We propose micromanagement tasks, which present the problem of the short-term, low-level control of army members during a battle. From a reinforcement learning point of view, these scenarios are challenging because the state-action space is very large, and because there is no obvious feature representation for the state-action evaluation function. We describe our approach to tackle the micromanagement scenarios with deep neural network controllers from raw state features given by the game engine. In addition, we present a heuristic reinforcement learning algorithm which combines direct exploration in the policy space and backpropagation. This algorithm allows for the collection of traces for learning using deterministic policies, which appears much more efficient than, for example, ε-greedy exploration. Experiments show that with this algorithm, we successfully learn non-trivial strategies for scenarios with armies of up to 15 agents, where both Q-learning and REINFORCE struggle.

Motivation & Objective

• Motivate reinforcement learning for real-time strategy micromanagement tasks with large state and action spaces and no obvious feature representation.
• Propose micromanagement tasks (e.g., m5v5, m15v16, dragoons_zealots, w15v17) and a deep neural network controller trained from raw game engine features.
• Develop a heuristic reinforcement learning algorithm that explores in policy space by randomizing network parameters at episode start and combining gradient-free updates with backpropagation.
• Evaluate standard RL baselines (Q-learning, REINFORCE) against the proposed approach on multi-unit StarCraft tasks, highlighting exploration and learning efficiency.

Proposed method

• Define micromanagement tasks in StarCraft with multiple units and durative actions (move and attack) and a greedy inference scheme to simplify joint action selection.
• Formulate the problem as an MDP with actions as sequences of unit-command pairs and introduce a greedy MDP that reduces joint action complexity while preserving optimality under sufficient policy capacity.
• Use a joint state-action feature representation with a deep neural network to score candidate commands for units, and apply a state-action embedding with pooling to handle variable unit counts.
• Introduce normalized cumulative rewards to stabilize learning across episodes with changing numbers of units, enabling consistent gradient updates.
• Present a zero-order backpropagation (ZO) algorithm that perturbs the deterministic policy together with gradient-based updates, mixing parameter space exploration with backpropagation.
• Employ Adagrad for parameter updates and rely on a two-stage embedding network that handles state-action features and action type (attack/move).

Experimental results

Research questions

• RQ1 Can RL algorithms learn meaningful micromanagement strategies in StarCraft with raw state features and no hand-crafted actions?
• RQ2 Does exploration in policy space via parameter randomization improve learning efficiency for deterministic policies in discrete action spaces?
• RQ3 Is greedy inference (sequential unit-wise action selection) effective for multi-unit control in StarCraft micromanagement tasks?
• RQ4 How do standard baselines (Q-learning, REINFORCE) perform compared to the proposed policy-space exploration method on multi-unit tasks?
• RQ5 What is the impact of reward normalization across varying unit counts on learning stability and performance?

Key findings

• The approach enables learning non-trivial strategies for scenarios with armies up to 15 units per side.
• Q-learning and REINFORCE struggle on these micromanagement tasks, while the proposed policy-space exploration method succeeds.
• Greedy inference with a deep state-action scorer can manage multi-unit coordination efficiently under deterministic policies.
• The zero-order backpropagation algorithm, which mixes parameter-space exploration with backpropagation, facilitates learning in large neural networks for this domain.
• Experiments demonstrate improvements over strong baselines and show the feasibility of learning micromanagement strategies directly from raw features.”