[Paper Review] Thinking Fast and Slow with Deep Learning and Tree Search

Thomas Anthony, Tian Zheng+1 more|arXiv (Cornell University)|May 23, 2017

Artificial Intelligence in GamesComputer Science15 references139 citations

TL;DR

Expert Iteration (ExIt) separates planning and learning by using tree search as an expert to train a neural network apprentice, which in turn guides search to improve future plans; the method achieves strong Hex performance and defeats MoHex.

ABSTRACT

Sequential decision making problems, such as structured prediction, robotic control, and game playing, require a combination of planning policies and generalisation of those plans. In this paper, we present Expert Iteration (ExIt), a novel reinforcement learning algorithm which decomposes the problem into separate planning and generalisation tasks. Planning new policies is performed by tree search, while a deep neural network generalises those plans. Subsequently, tree search is improved by using the neural network policy to guide search, increasing the strength of new plans. In contrast, standard deep Reinforcement Learning algorithms rely on a neural network not only to generalise plans, but to discover them too. We show that ExIt outperforms REINFORCE for training a neural network to play the board game Hex, and our final tree search agent, trained tabula rasa, defeats MoHex 1.0, the most recent Olympiad Champion player to be publicly released.

Motivation & Objective

• Address sequential decision making by combining planning with function approximation.
• Propose Expert Iteration (ExIt) to separate expert planning and apprentice generalisation.
• Show that neural networks guided by planning improve search and learning efficiency.
• Demonstrate ExIt on Hex, comparing with REINFORCE and MoHex to establish competitiveness.

Proposed method

• Define Expert Iteration (ExIt) as an iterative loop: self-play to collect states, imitation learning to train an apprentice, and expert improvement via tree search guided by the apprentice.
• Treat the expert as a tree search algorithm and the apprentice as a deep neural network policy (and optionally value network).
• Use imitation learning targets (CAT and tree-policy targets, TPT) to train the apprentice from the expert’s moves; TPT is cost-sensitive.
• Employ online dataset aggregation (DAgger-like) to improve data efficiency and reduce recomputation.
• Bias the tree search by the apprentice policy through an adjusted UCT formula with a bonus term.
• Extend the framework to incorporate value networks to estimate leaf values and blend them with rollouts.
• Demonstrate online distributed ExIt and compare batch vs online variants against REINFORCE and MoHex.

Experimental results

Research questions

• RQ1 Can ExIt learn stronger policies faster than standard policy gradient methods (e.g., REINFORCE) in Hex?
• RQ2 Does separating planning (expert) and generalisation (apprentice) improve learning efficiency and final performance?
• RQ3 How do online (dataset-aggregated) versus batch ExIt compare in data efficiency and stability?
• RQ4 What is the impact of using policy-only versus policy-and-value networks in ExIt on search strength and performance against a strong baseline like MoHex?

Key findings

• ExIt outperforms REINFORCE in training a neural network to play Hex.
• A final tabula rasa ExIt agent defeats MoHex 1.0 in head-to-head play.
• Tree-policy targets (TPT) yield stronger performance than chosen-action targets (CAT) during imitation learning (50±13 Elo improvement reported after initial data).
• DAgger-style online ExIt with dataset aggregation improves data efficiency and final strength over batch ExIt.
• Using a policy network to bias MCTS (Neural-MCTS) dramatically increased win rates (e.g., 97% wins with a strong policy network vs baseline MCTS).
• Adding a value network to the apprentice significantly improves expert quality over policy-only ExIt, leading to stronger play.