[Paper Review] StarCraft II: A New Challenge for Reinforcement Learning

Oriol Vinyals, Timo Ewalds+23 more|arXiv (Cornell University)|Aug 16, 2017

Digital Games and Media11 references683 citations

TL;DR

The paper introduces SC2LE (StarCraft II Learning Environment), an RTS-based RL benchmark with full-game and mini-game tasks, outlining its observation/action/reward interfaces and presenting baseline RL results. It argues SC2LE as a challenging, multi-agent, partially observable domain for advancing deep RL architectures.

ABSTRACT

This paper introduces SC2LE (StarCraft II Learning Environment), a reinforcement learning environment based on the StarCraft II game. This domain poses a new grand challenge for reinforcement learning, representing a more difficult class of problems than considered in most prior work. It is a multi-agent problem with multiple players interacting; there is imperfect information due to a partially observed map; it has a large action space involving the selection and control of hundreds of units; it has a large state space that must be observed solely from raw input feature planes; and it has delayed credit assignment requiring long-term strategies over thousands of steps. We describe the observation, action, and reward specification for the StarCraft II domain and provide an open source Python-based interface for communicating with the game engine. In addition to the main game maps, we provide a suite of mini-games focusing on different elements of StarCraft II gameplay. For the main game maps, we also provide an accompanying dataset of game replay data from human expert players. We give initial baseline results for neural networks trained from this data to predict game outcomes and player actions. Finally, we present initial baseline results for canonical deep reinforcement learning agents applied to the StarCraft II domain. On the mini-games, these agents learn to achieve a level of play that is comparable to a novice player. However, when trained on the main game, these agents are unable to make significant progress. Thus, SC2LE offers a new and challenging environment for exploring deep reinforcement learning algorithms and architectures.

Motivation & Objective

• Introduce SC2LE as a reinforcement learning environment based on StarCraft II.
• Characterize the domain challenges: multi-agent interaction, imperfect information, large action/state spaces, and long-term credit assignment.
• Provide an open-source interface (PySC2) and datasets (human replays) for RL research.
• Offer baseline results to calibrate difficulty and guide future RL algorithm development.

Proposed method

• Define observations as low-resolution feature layers and supplementary non-spatial data.
• Design an action space that mirrors human UI with ~300 action-function identifiers and 13 argument types.
• Employ Asynchronous Advantage Actor-Critic (A3C) with n-step returns and entropy regularization as the baseline learning algorithm.
• Evaluate multiple neural architectures (Atari-net like, FullyConv, and FullyConv with LSTM) to map observations to action policies.
• Provide mini-game tasks with tailored rewards to isolate specific gameplay elements.

Experimental results

Research questions

• RQ1 Can deep RL agents learn meaningful policies for full StarCraft II games using SC2LE interfaces?
• RQ2 Do standard RL baselines (A3C) scale to the large action/state spaces of StarCraft II?
• RQ3 How do different neural architectures perform on the SC2LE observations, including spatially-aware networks?
• RQ4 What is the value of mini-games for isolating and solving sub-tasks within StarCraft II?
• RQ5 How does agent performance differ when trained on full games versus mini-games or random baselines?

Key findings

• BaselineRL agents struggle to win the full game against easy AI on ladder maps.
• Agents trained with Blizzard score rewards converge to simple mining-focused or non-advancing strategies.
• A fully convolutional, memory-enabled architecture shows more robust behavior but still fails to achieve winning performance on the full game.
• Mini-games allow agents to reach novice-level play, but full-game progress remains limited under the tested baselines.
• The SC2LE setup yields a challenging benchmark for advancing deep RL architectures, perception, memory, and decision-making in complex environments.