[Paper Review] Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

David Silver, Thomas Hubert+11 more|arXiv (Cornell University)|Dec 5, 2017

Artificial Intelligence in Games23 references1,079 citations

TL;DR

AlphaZero general reinforcement learning algorithm learns chess, shogi, and Go from scratch via self-play, using a single neural network and Monte-Carlo tree search to reach superhuman performance in hours. It defeats world-champion programs in each domain.

ABSTRACT

The game of chess is the most widely-studied domain in the history of artificial intelligence. The strongest programs are based on a combination of sophisticated search techniques, domain-specific adaptations, and handcrafted evaluation functions that have been refined by human experts over several decades. In contrast, the AlphaGo Zero program recently achieved superhuman performance in the game of Go, by tabula rasa reinforcement learning from games of self-play. In this paper, we generalise this approach into a single AlphaZero algorithm that can achieve, tabula rasa, superhuman performance in many challenging domains. Starting from random play, and given no domain knowledge except the game rules, AlphaZero achieved within 24 hours a superhuman level of play in the games of chess and shogi (Japanese chess) as well as Go, and convincingly defeated a world-champion program in each case.

Motivation & Objective

• Demonstrate that a single general-purpose RL algorithm can achieve tabula rasa, superhuman performance across multiple complex board games without domain knowledge beyond rules.
• Show that self-play with neural networks and MCTS can match or exceed specialized engines in chess and shogi, as well as Go.
• Analyze the learning dynamics, search efficiency, and scalability of the approach compared to traditional domain-specific methods.
• Reveal whether the learned strategies in chess include human-like openings and whether the method generalizes across games.

Proposed method

• Use a general AlphaZero algorithm to learn via self-play from random play with no domain knowledge beyond game rules.
• Represent board states and actions as neural network inputs/outputs tuned to each game, including a policy and value head f_theta(s) -> (p,v).
• Train the network through self-play using MCTS guided by the network, with a loss combining mean-squared error and cross-entropy to align v and p with outcomes and search probabilities.
• Maintain a single continuously updated network rather than iterating against a best-so-far player, and apply Dirichlet noise to the root prior for exploration.
• Apply the same algorithm settings across Go, chess, and shogi, with game-specific input/output representations and 800 MCTS simulations per move.
• Evaluate performance against strongest domain-specific engines (Stockfish for chess, Elmo for shogi) and against AlphaGo Zero predecessors in Go.

Experimental results

Research questions

• RQ1 Can a general-purpose reinforcement learning algorithm achieve superhuman performance in chess and shogi starting from random play (tabula rasa)?
• RQ2 How does AlphaZero's MCTS guided by a neural network compare to traditional alpha-beta search engines in these domains?
• RQ3 Does the learned policy explore human-like openings and strategies in chess, and do these general methods transfer across games?
• RQ4 How does the learning curve and time-to-superhuman performance scale with training for different games?

Key findings

• AlphaZero surpasses Stockfish in chess after about 300k steps (4 hours) using self-play from random play.
• AlphaZero surpasses Elmo in shogi after about 110k steps (less than 2 hours).
• AlphaZero defeats the previous 3-day AlphaGo Zero version in Go after about 165k steps (8 hours).
• In 100-game tournaments, AlphaZero defeats Stockfish, Elmo, and the 3-day AlphaGo Zero, without losing a game against Stockfish and with only a handful of losses to Elmo or AG0 variants.
• AlphaZero’s MCTS with neural networks scales more effectively with thinking time than the baseline alpha-beta engines tested, challenging assumptions about search superiority in these domains.
• AlphaZero learns and frequently employs common human openings during self-play, and starting from these openings, it defeats Stockfish.