[Paper Review] Hyper-Parameter Sweep on AlphaZero General

Hui Wang, Michael Emmerich+2 more|arXiv (Cornell University)|Mar 19, 2019

Artificial Intelligence in Games24 references8 citations

TL;DR

This paper conducts a comprehensive hyper-parameter sweep on AlphaZeroGeneral, evaluating 12 key parameters across training loss, Elo rating, and time cost in 6×6 Othello. It identifies time-sensitive vs. time-friendly parameters, reveals non-monotonic performance trends (e.g., optimal values exist beyond maximums), and provides a data-driven basis for hyper-parameter optimization in AlphaZero-based agents.

ABSTRACT

Since AlphaGo and AlphaGo Zero have achieved breakground successes in the game of Go, the programs have been generalized to solve other tasks. Subsequently, AlphaZero was developed to play Go, Chess and Shogi. In the literature, the algorithms are explained well. However, AlphaZero contains many parameters, and for neither AlphaGo, AlphaGo Zero nor AlphaZero, there is sufficient discussion about how to set parameter values in these algorithms. Therefore, in this paper, we choose 12 parameters in AlphaZero and evaluate how these parameters contribute to training. We focus on three objectives~(training loss, time cost and playing strength). For each parameter, we train 3 models using 3 different values~(minimum value, default value, maximum value). We use the game of play 6$ imes$6 Othello, on the AlphaZeroGeneral open source re-implementation of AlphaZero. Overall, experimental results show that different values can lead to different training results, proving the importance of such a parameter sweep. We categorize these 12 parameters into time-sensitive parameters and time-friendly parameters. Moreover, through multi-objective analysis, this paper provides an insightful basis for further hyper-parameter optimization.

Motivation & Objective

• To address the lack of systematic parameter tuning guidance in AlphaZero and related algorithms, especially for hyper-parameters not well-documented in original DeepMind publications.
• To evaluate the impact of 12 core hyper-parameters on three key objectives: training loss, playing strength (Elo rating), and computational time cost.
• To classify parameters as time-sensitive or time-friendly based on their influence on training duration, enabling more efficient optimization strategies.
• To provide a data-driven, empirical basis for hyper-parameter selection in AlphaZeroGeneral, supporting transferability to other games and tasks.

Proposed method

• The study uses a lightweight open-source re-implementation of AlphaZero, AlphaZeroGeneral, to train 3 models per parameter using three values: minimum, default, and maximum.
• Each experiment isolates one parameter while holding all others at their default values, ensuring controlled evaluation.
• Three evaluation metrics are used: mean training loss over 1000 steps, final Elo rating after 1000 games, and total training time in hours.
• A time cost function is derived empirically (Equation 2), enabling quantification of computational overhead per parameter setting.
• Parameters are categorized as time-sensitive or time-friendly based on their impact on training duration, with statistical analysis of performance trends.
• Multi-objective analysis is performed to identify Pareto-optimal configurations balancing loss, strength, and efficiency.

Experimental results

Research questions

• RQ1 How do different values of 12 hyper-parameters affect the training loss, Elo rating, and time cost in AlphaZeroGeneral?
• RQ2 Which hyper-parameters are time-sensitive (significantly affecting training time) versus time-friendly (minimal impact on time)?
• RQ3 Does increasing a parameter value always improve performance, or is there an optimal value beyond which performance degrades?
• RQ4 Can a multi-objective analysis identify parameter configurations that balance training efficiency, loss reduction, and playing strength?
• RQ5 What are the most promising hyper-parameter settings for minimizing training loss, maximizing Elo rating, and reducing time cost?

Key findings

• The parameter 'mctssimu' (MCTS simulations per move) has a strong positive impact on Elo rating, with 200 simulations yielding the highest performance, though it increases time cost significantly.
• The 'iteration' parameter (number of self-play games per training phase) shows a non-monotonic relationship with performance: the default value of 100 achieves the best balance between loss and Elo rating, while higher values (e.g., 200) degrade performance.
• The 'batchsize' parameter has a strong negative correlation with time cost: larger batch sizes (e.g., 64) reduce training time by 16% compared to smaller sizes (e.g., 32), making it highly time-sensitive.
• The 'updateThreshold' parameter, which controls when to update the neural network policy, achieves peak Elo rating at 0.6, indicating a non-trivial optimal value rather than a monotonic improvement with higher values.
• Parameters like 'Cpuct', 'learningrate', 'dropout', and 'tempThreshold' are classified as time-friendly, as their variation causes minimal change in training duration, enabling safe tuning for performance.
• The 'arenacompare' parameter (number of games in arena comparison) shows minimal impact on both Elo rating and time cost, suggesting it can be tuned freely without performance or efficiency trade-offs.