[论文解读] Accelerating Self-Play Learning in Go

David Wu|arXiv (Cornell University)|Feb 27, 2019

Artificial Intelligence in Games参考文献 19被引用 49

一句话总结

KataGo 在围棋领域实现约50x 的计算效率，并在硬件资源更少的情况下超越 ELF OpenGo，具备领域无关与领域特定的改进。

ABSTRACT

By introducing several improvements to the AlphaZero process and architecture, we greatly accelerate self-play learning in Go, achieving a 50x reduction in computation over comparable methods. Like AlphaZero and replications such as ELF OpenGo and Leela Zero, our bot KataGo only learns from neural-net-guided Monte Carlo tree search self-play. But whereas AlphaZero required thousands of TPUs over several days and ELF required thousands of GPUs over two weeks, KataGo surpasses ELF's final model after only 19 days on fewer than 30 GPUs. Much of the speedup involves non-domain-specific improvements that might directly transfer to other problems. Further gains from domain-specific techniques reveal the remaining efficiency gap between the best methods and purely general methods such as AlphaZero. Our work is a step towards making learning in state spaces as large as Go possible without large-scale computational resources.

研究动机与目标

• Motivate reducing computational resources required for self-play learning in Go without external human data or knowledge.
• Develop general improvements transferable to AlphaZero-like reinforcement learning and identify remaining efficiency gaps.
• Demonstrate domain-specific techniques that further accelerate Go learning beyond general methods.

提出的方法

• Adopt a planner-augmented MCTS with neural-net guided search resembling AlphaGo/Zero architectures.
• Introduce playout cap randomization to balance policy and value training by varying search depth, with full searches on a subset of turns.
• Implement policy target pruning to decouple exploration from policy targets and enforce forced playouts.
• Add global pooling to neural nets to provide global context across the board state.
• Incorporate auxiliary policy targets predicting opponent replies to regularize training.
• Integrate domain-specific features and ownership/score targets to improve learning efficiency.

实验结果

研究问题

• RQ1 Can non-domain-specific improvements alone close the efficiency gap relative to AlphaZero-like methods?
• RQ2 How much do domain-specific features (ownership, score targets) contribute to learning efficiency in Go?
• RQ3 What is the impact of techniques like playout cap randomization, policy target pruning, and global pooling on sample efficiency and final strength?
• RQ4 How does KataGo perform relative to ELF OpenGo and Leela Zero under comparable compute budgets?
• RQ5 To what extent can auxiliary targets and input features generalize to reinforcement learning tasks beyond Go?

主要发现

组件	Elo	因子
主运行，基线	1329	1.00x
放局上限随机化	1242	1.37x
强制下棋与策略目标剪枝	1276	1.25x
全局池化	1153	1.60x
辅助策略目标	1255	1.30x
辅助所有者与分数目标	1139	1.65x
游戏特定特征与选项	1168	1.55x

• KataGo achieved competitive strength with ~1.4 GPU-years on 27 GPUs over 19 days, ~50x more efficient than ELF/OpenGo scales.
• Against ELF, KataGo outperformed with a ~50x efficiency advantage in self-play compute (relative Elo gains demonstrated).
• Ablation experiments show playout cap randomization, global pooling, and auxiliary targets each provide measurable efficiency gains, with combined effects yielding substantial speedups (approximate factor sums in Table 2).
• Auxiliary ownership and score targets significantly improved learning efficiency; Go-specific input features also contributed meaningfully to speedups beyond general methods.
• Policy target pruning with forced playouts decouples policy targets from search dynamics, aiding convergence of the neural net.
• KataGo demonstrates that a large efficiency gap remains between AlphaZero-like methods and optimized self-play, even for Go, suggesting room for further data-efficient improvements.