[Paper Review] Monte-Carlo Tree Search as Regularized Policy Optimization

Jean-Bastien Grill, Florent Altché+5 more|arXiv (Cornell University)|Jul 24, 2020

Reinforcement Learning in Robotics34 citations

TL;DR

The paper shows that AlphaZero-like MCTS heuristics approximate a regularized policy optimization problem and proposes a variant using the exact solution, yielding improved performance, especially at low simulation budgets.

ABSTRACT

The combination of Monte-Carlo tree search (MCTS) with deep reinforcement learning has led to significant advances in artificial intelligence. However, AlphaZero, the current state-of-the-art MCTS algorithm, still relies on handcrafted heuristics that are only partially understood. In this paper, we show that AlphaZero's search heuristics, along with other common ones such as UCT, are an approximation to the solution of a specific regularized policy optimization problem. With this insight, we propose a variant of AlphaZero which uses the exact solution to this policy optimization problem, and show experimentally that it reliably outperforms the original algorithm in multiple domains.

Motivation & Objective

• Motivate the connection between MCTS (especially AlphaZero) and regularized policy optimization (MPO).
• Demonstrate that AlphaZero’s search distributions approximate a regularized policy optimization solution.
• Propose and evaluate a variant of AlphaZero that uses the exact MPO solution to improve robustness and performance.

Proposed method

• Define empirical visit distribution hat{
  d} and its relation to a regularized objective.
• Show that AlphaZero's action selection approximates the solution to a regularized policy optimization with KL-divergence regularization.
• Derive the exact MPO-like solution ar{
  d} and compare it to the empirical visit distribution.
• Propose three variants (Act, Search, Learn) that replace ha0 with ar{
  d} in different components of AlphaZero.
• Provide an All variant combining Act, Search, Learn and show its training target uses ar{
  d} for learning.
• Relate AlphaZero and UCT to regularized policy optimization and discuss broader implications for MCTS-based methods.

Experimental results

Research questions

• RQ1 Does AlphaZero’s search policy correspond to a regularized policy optimization solution?
• RQ2 Can replacing the empirical visit distribution with the exact MPO-like solution improve performance, especially under low simulation budgets?
• RQ3 How do actions sampling from the MPO-like policy affect learning, search, and acting components in MCTS-based agents?
• RQ4 Do these insights extend to other MCTS variants like UCT with learned priors?
• RQ5 Do the proposed variants yield empirical gains in Atari and continuous control domains?

Key findings

• AlphaZero and similar MCTS methods approximate the solution to a regularized policy optimization problem.
• The MPO-like solution ar{
  d} can be computed and used as a more stable target than the empirical visit distribution, particularly at low simulation budgets.
• An All variant using ar{
  d} for acting, searching, and learning outperforms the baseline MuZero in Atari Ms. Pac-Man and continuous control tasks under low simulation budgets.
• The performance advantage of ar{
  d}-based approaches is most pronounced at small N_sim, with benefits diminishing as simulation budgets grow, aligning with theoretical convergence.
• Using ar{
  d} during search yields notable gains, while using it for learning helps at very low budgets; combining approaches yields robust improvements across tasks.