[Paper Review] Asynchronous Advantage Actor-Critic Agent for Starcraft II

Basel Alghanem, P G Keerthana|arXiv (Cornell University)|Jul 22, 2018

Reinforcement Learning in Robotics7 references4 citations

TL;DR

This paper proposes an Asynchronous Advantage Actor-Critic (A3C) agent for StarCraft II using transfer learning to improve sample efficiency and convergence speed across multiple minigames. The authors demonstrate that pre-training on simpler scenarios significantly accelerates learning and enhances performance on complex, composite maps like Simple64, outperforming training from scratch.

ABSTRACT

Deep reinforcement learning, and especially the Asynchronous Advantage Actor-Critic algorithm, has been successfully used to achieve super-human performance in a variety of video games. Starcraft II is a new challenge for the reinforcement learning community with the release of pysc2 learning environment proposed by Google Deepmind and Blizzard Entertainment. Despite being a target for several AI developers, few have achieved human level performance. In this project we explain the complexities of this environment and discuss the results from our experiments on the environment. We have compared various architectures and have proved that transfer learning can be an effective paradigm in reinforcement learning research for complex scenarios requiring skill transfer.

Motivation & Objective

• To evaluate the effectiveness of different neural network architectures in training deep reinforcement learning agents for StarCraft II.
• To investigate whether transfer learning can accelerate learning and improve policy performance across different StarCraft II minigames.
• To analyze the trade-off between training speed and convergence to suboptimal policies in complex, high-dimensional environments.
• To determine the impact of reward sparsity and partial observability on agent training and policy learning.
• To explore the feasibility of using pre-trained policies from simpler tasks to solve more complex composite scenarios.

Proposed method

• The authors implement an Asynchronous Advantage Actor-Critic (A3C) algorithm to train agents on the PySC2 environment, leveraging parallel asynchronous training on multiple environments.
• They evaluate three network architectures: a baseline feedforward network, a fully connected network with residual connections (PlusFC), and a convolutional network (PlusConv), with the latter two incorporating residual and convolutional operations for better feature extraction.
• Transfer learning is applied by initializing the policy network for a new task using a pre-trained model from a related but simpler task, such as using a FindAndDefeatZerglings policy to initialize training on DefeatZerglingsAndBanelings.
• The agents are trained using both sparse ternary rewards (win/loss/tie) and dense Blizzard scores, which provide continuous feedback and improve learning stability.
• Training is conducted on a suite of StarCraft II minigames, including CollectMineralShards, FindAndDefeatZerglings, DefeatZerglingsAndBanelings, BuildMarines, and the composite Simple64 map.
• Model pruning and checkpoint restoration are used to mitigate policy degradation during training, especially in high-difficulty scenarios.

Experimental results

Research questions

• RQ1 Can transfer learning significantly reduce training time and improve performance in complex StarCraft II minigames?
• RQ2 How do different neural network architectures (baseline, PlusFC, PlusConv) compare in terms of convergence speed and final performance on StarCraft II tasks?
• RQ3 To what extent does the use of dense Blizzard scores improve learning compared to sparse ternary rewards?
• RQ4 Does pre-training on a simpler task lead to better policy generalization on more complex, composite scenarios like Simple64?
• RQ5 Why do agents often converge to suboptimal policies in high-difficulty environments, and can architectural or algorithmic modifications mitigate this?

Key findings

• Transfer learning using a pre-trained policy from FindAndDefeatZerglings significantly accelerated training and improved performance on the DefeatZerglingsAndBanelings task, with scores improving over time as the agent explored new, more effective strategies.
• The PlusConv and PlusFC architectures outperformed the baseline network in terms of both convergence speed and final performance, particularly in complex scenarios requiring spatial reasoning and long-horizon planning.
• Training from scratch on the BuildMarines minigame failed to produce any meaningful progress within 4,000 episodes, as the agent never learned to build marines, resulting in zero rewards, highlighting the challenge of sparse rewards and high action space complexity.
• On the Simple64 composite map, the agent achieved non-zero scores after 18,000 episodes using the baseline network, though performance remained below leaderboard levels, suggesting room for improvement in multi-task generalization.
• The agent trained on FindAndDefeatZerglings showed improved exploration over time, but still left 30–40% of the map unexplored, indicating that exploration remains a key challenge in partially observable environments.
• Policy degradation during training necessitated model pruning and retraining from good checkpoints, underscoring the instability of training in high-dimensional, sparse-reward environments.