[Paper Review] Watch the Unobserved: A Simple Approach to Parallelizing Monte Carlo Tree Search

Anji Liu, Jianshu Chen+4 more|arXiv (Cornell University)|Oct 28, 2018

Artificial Intelligence in Games35 references4 citations

TL;DR

This paper proposes WU-UCT, a novel parallel Monte Carlo Tree Search (MCTS) algorithm that achieves linear speedup by tracking unobserved (incomplete) simulation queries via a new statistics mechanism. By modifying the UCT tree policy to incorporate these unobserved samples, WU-UCT maintains effective exploration-exploitation tradeoff during parallel rollouts, significantly reducing performance loss compared to prior parallel MCTS methods.

ABSTRACT

Monte Carlo Tree Search (MCTS) algorithms have achieved great success on many challenging benchmarks (e.g., Computer Go). However, they generally require a large number of rollouts, making their applications costly. Furthermore, it is also extremely challenging to parallelize MCTS due to its inherent sequential nature: each rollout heavily relies on the statistics (e.g., node visitation counts) estimated from previous simulations to achieve an effective exploration-exploitation tradeoff. In spite of these difficulties, we develop an algorithm, WU-UCT, to effectively parallelize MCTS, which achieves linear speedup and exhibits only limited performance loss with an increasing number of workers. The key idea in WU-UCT is a set of statistics that we introduce to track the number of on-going yet incomplete simulation queries (named as unobserved samples). These statistics are used to modify the UCT tree policy in the selection steps in a principled manner to retain effective exploration-exploitation tradeoff when we parallelize the most time-consuming expansion and simulation steps. Experiments on a proprietary benchmark and the Atari Game benchmark demonstrate the linear speedup and the superior performance of WU-UCT comparing to existing techniques.

Motivation & Objective

• To address the challenge of parallelizing MCTS, which is inherently sequential due to reliance on cumulative statistics for exploration-exploitation tradeoff.
• To reduce performance degradation in parallel MCTS caused by loss of information from incomplete or concurrent rollouts.
• To develop a practical and efficient parallel MCTS algorithm that achieves near-linear speedup without sacrificing decision quality.
• To enable scalable, high-performance planning in real-world applications such as game AI and user behavior prediction in mobile games.

Proposed method

• Introduces a new set of statistics to track the number of on-going, incomplete simulation queries—termed 'unobserved samples'—to preserve information during parallelization.
• Modifies the UCT tree policy in the selection phase by incorporating both observed and unobserved sample statistics to maintain balanced exploration and exploitation.
• Employs a virtual loss mechanism in tree traversal to prevent race conditions and ensure consistent policy updates during asynchronous rollouts.
• Uses a hybrid parallelization strategy combining tree and root parallelism, with dynamic workload distribution across workers to balance load.
• Implements a backpropagation mechanism that accounts for unobserved samples by adjusting visit counts and value estimates in a principled way.
• Deploys the algorithm in production for user pass-rate prediction in the mobile game 'Joy City', validating real-world scalability and accuracy.

Experimental results

Research questions

• RQ1 Can we achieve linear speedup in MCTS while minimizing performance degradation due to parallelization?
• RQ2 How can we preserve the exploration-exploitation tradeoff when multiple rollouts are executed concurrently?
• RQ3 What statistics are needed to account for incomplete simulations in a parallel MCTS setting?
• RQ4 Can a simple, principled modification to the UCT policy maintain performance under high parallelism?
• RQ5 How does the proposed method compare to existing parallel MCTS techniques in real-world and benchmark environments?

Key findings

• WU-UCT achieves near-linear speedup on both the proprietary 'Joy City' benchmark and the Atari Game benchmark, with minimal performance loss as the number of workers increases.
• On the 'Joy City' benchmark, WU-UCT outperforms existing parallel MCTS methods in predicting user pass-rates, demonstrating superior accuracy and efficiency.
• The algorithm maintains a stable and effective exploration-exploitation tradeoff even with high parallelism, thanks to the unobserved sample tracking mechanism.
• Experiments show that WU-UCT's performance loss is significantly lower than that of baseline methods like TreeP and RootP, especially at scale.
• The virtual loss mechanism successfully prevents overestimation of action values during concurrent rollouts, improving convergence and stability.
• The method is successfully deployed in production, reducing design cycle time for level progression in a real mobile game.