[Paper Review] Train on Small, Play the Large: Scaling Up Board Games with AlphaZero and GNN

Shai Ben-Assayag, Ran El‐Yaniv|arXiv (Cornell University)|Jul 18, 2021

Educational Games and Gamification27 references4 citations

TL;DR

This paper proposes ScalableAlphaZero (SAZ), a reinforcement learning framework that combines Graph Neural Networks (GNNs) with the AlphaZero algorithm to train on small board sizes and generalize to larger ones without retraining. By replacing convolutional networks with GNNs and introducing subgraph sampling to reduce prediction uncertainty, SAZ achieves performance comparable to AlphaZero on large boards—training in just three days versus 30 days—demonstrating a 10x reduction in training time with superior scalability.

ABSTRACT

Playing board games is considered a major challenge for both humans and AI researchers. Because some complicated board games are quite hard to learn, humans usually begin with playing on smaller boards and incrementally advance to master larger board strategies. Most neural network frameworks that are currently tasked with playing board games neither perform such incremental learning nor possess capabilities to automatically scale up. In this work, we look at the board as a graph and combine a graph neural network architecture inside the AlphaZero framework, along with some other innovative improvements. Our ScalableAlphaZero is capable of learning to play incrementally on small boards, and advancing to play on large ones. Our model can be trained quickly to play different challenging board games on multiple board sizes, without using any domain knowledge. We demonstrate the effectiveness of ScalableAlphaZero and show, for example, that by training it for only three days on small Othello boards, it can defeat the AlphaZero model on a large board, which was trained to play the large board for $30$ days.

Motivation & Objective

• To enable deep reinforcement learning agents to learn scalable board game strategies across multiple board sizes without retraining from scratch.
• To overcome the limitation of AlphaZero’s fixed-size input requirement by replacing its convolutional neural network with a graph neural network (GNN) for inherent scalability.
• To reduce prediction uncertainty in GNN-based value and policy networks through a novel subgraph sampling technique that enhances model robustness.
• To demonstrate that training on small boards can yield agents competitive with those trained extensively on large boards, significantly reducing computational cost.

Proposed method

• Replace the convolutional neural network (CNN) in AlphaZero with a Graph Neural Network (GNN), enabling the model to process variable-sized board inputs without architectural changes.
• Use a GNN to encode the board as a graph, where each intersection is a node and adjacent intersections are connected by edges, allowing the model to learn spatial dependencies across any board size.
• Introduce subgraph sampling: sample multiple subgraphs from the full board, process each with the same GNN, and average or ensemble their predictions to reduce uncertainty and improve robustness.
• Integrate the GNN-based value and policy networks into the AlphaZero framework, using Monte Carlo Tree Search (MCTS) with self-play for policy improvement.
• Train the model exclusively on small boards (e.g., 9×9) and evaluate its performance on larger boards (e.g., 16×16, 20×20) without any fine-tuning on the larger size.
• Use a shared GNN architecture across all board sizes, maintaining a constant number of parameters regardless of input dimension.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent trained on small board sizes generalize effectively to much larger board sizes in scalable games like Othello, Gomoku, and Go?
• RQ2 Does replacing the CNN in AlphaZero with a GNN enable true scalability across board sizes while preserving or improving performance?
• RQ3 To what extent does subgraph sampling reduce prediction uncertainty in GNN-based agents and improve generalization in board game environments?
• RQ4 Can a model trained for only three days on small boards outperform an AlphaZero agent trained for 30 days on large boards, despite no direct training on the large board?
• RQ5 How does the performance of the scalable model compare to standard AlphaZero when tested on large boards, especially in terms of win rate and computational efficiency?

Key findings

• ScalableAlphaZero (SAZ) trained for only three days on 9×9 Othello boards achieved a 54% win rate against an AlphaZero agent trained for 30 days on a 16×16 board.
• On 20×20 Othello boards, SAZ won 84% of games against the same 30-day AlphaZero agent, demonstrating strong generalization to larger boards.
• On 19×19 Gomoku, SAZ achieved a 100% win rate against the AlphaZero model trained on the same large board, despite no direct training on that size.
• The ablation study showed that removing the subgraph sampling technique reduced performance by 24% on Othello, confirming its critical role in uncertainty reduction.
• On Go, SAZ trained for three days on 9×9 boards won 68% of games against an AlphaZero agent trained for 20 days on 9×9, and 77.5% against a 10-day 15×15 AlphaZero model.
• The model without subgraph sampling (model4) performed poorly on Othello (28% win rate) but improved on Gomoku (98%), indicating that global board structure is more critical than local patterns in certain games.