[Paper Review] StarCraft II Build Order Optimization using Deep Reinforcement Learning and Monte-Carlo Tree Search

Islam Elnabarawy, Kristijana Arroyo+1 more|arXiv (Cornell University)|Jun 12, 2020

Artificial Intelligence in Games7 references4 citations

TL;DR

This paper proposes a hybrid approach combining Monte-Carlo Tree Search (MCTS) and deep reinforcement learning to optimize StarCraft II build orders, specifically in the 'BuildMarines' minigame. The MCTS agent achieves novice-level performance with minimal computation, and the integration of a deep neural network is shown to be essential for reaching expert-level results by reducing rollout costs and enabling deeper search within time constraints.

ABSTRACT

The real-time strategy game of StarCraft II has been posed as a challenge for reinforcement learning by Google's DeepMind. This study examines the use of an agent based on the Monte-Carlo Tree Search algorithm for optimizing the build order in StarCraft II, and discusses how its performance can be improved even further by combining it with a deep reinforcement learning neural network. The experimental results accomplished using Monte-Carlo Tree Search achieves a score similar to a novice human player by only using very limited time and computational resources, which paves the way to achieving scores comparable to those of a human expert by combining it with the use of deep reinforcement learning.

Motivation & Objective

• To investigate the feasibility of using Monte-Carlo Tree Search (MCTS) for optimizing build orders in StarCraft II.
• To evaluate the performance of MCTS in isolation on the 'BuildMarines' minigame, a simplified environment for macro-strategy learning.
• To identify the limitations of pure MCTS in achieving expert-level performance due to computational constraints.
• To lay the foundation for combining MCTS with deep reinforcement learning to significantly improve optimization efficiency and outcome quality.

Proposed method

• The agent uses an abstract state representation based on key game variables: time to produce supply depot, barracks, and marine.
• MCTS is applied using the UCB1 selection strategy with an exploration parameter β = 1/√2.
• Actions include no-op, train worker, build supply, build barracks, and train marine, with state transitions updated based on action selection.
• Production queue times advance each frame, and mineral collection is simulated at a rate proportional to the number of workers.
• The game ends after ~10 minutes, with the final score being the number of marines alive at termination.
• The MCTS agent performs multiple iterations per time step, with results collected across different iteration limits (1 to 10).

Experimental results

Research questions

• RQ1 Can MCTS alone achieve competitive build order optimization in the StarCraft II 'BuildMarines' minigame?
• RQ2 How does increasing MCTS iteration count affect performance in terms of final marine count?
• RQ3 What is the performance ceiling of pure MCTS in this environment, and how does it compare to novice and expert human players?
• RQ4 To what extent can a deep reinforcement learning neural network improve MCTS performance by replacing costly rollouts with value function estimation?

Key findings

• The MCTS agent achieved a peak score of 94 marines with 5 iterations, which is comparable to the performance of a novice human player.
• Scores plateaued around 90–94 marines for iteration counts between 3 and 10, indicating diminishing returns from increased iterations.
• Scores did not significantly improve beyond 94 marines within the tested iteration range, suggesting computational limits of pure MCTS.
• The results indicate that achieving scores above 100 marines requires a substantial increase in computation time, implying that pure MCTS is insufficient for expert-level performance.
• The study concludes that integrating a deep reinforcement learning neural network is necessary to reduce rollout cost and enable deeper, more efficient search within fixed time constraints.
• The combination of MCTS and deep reinforcement learning is expected to enable the agent to reach performance levels comparable to expert human players.