[Paper Review] Non-stochastic Best Arm Identification and Hyperparameter Optimization

Kevin Jamieson, Ameet Talwalkar|arXiv (Cornell University)|Feb 27, 2015

Advanced Bandit Algorithms ResearchDecision Sciences20 references237 citations

TL;DR

This paper introduces a non-stochastic best-arm identification framework for hyperparameter optimization, where arms represent hyperparameter configurations and losses are observed over time during training. By leveraging the Successive Halving algorithm, the method achieves comparable model performance an order of magnitude faster than baselines by adaptively allocating resources to promising configurations, with empirical results showing significant speedups in wall-clock time across ridge regression, kernel SVM, and matrix completion tasks.

ABSTRACT

Motivated by the task of hyperparameter optimization, we introduce the non-stochastic best-arm identification problem. Within the multi-armed bandit literature, the cumulative regret objective enjoys algorithms and analyses for both the non-stochastic and stochastic settings while to the best of our knowledge, the best-arm identification framework has only been considered in the stochastic setting. We introduce the non-stochastic setting under this framework, identify a known algorithm that is well-suited for this setting, and analyze its behavior. Next, by leveraging the iterative nature of standard machine learning algorithms, we cast hyperparameter optimization as an instance of non-stochastic best-arm identification, and empirically evaluate our proposed algorithm on this task. Our empirical results show that, by allocating more resources to promising hyperparameter settings, we typically achieve comparable test accuracies an order of magnitude faster than baseline methods.

Motivation & Objective

• Address the lack of theoretical and algorithmic frameworks for best-arm identification in non-stochastic multi-armed bandit settings, particularly for hyperparameter optimization.
• Overcome the limitations of existing methods that treat model training as a black box or rely on strong assumptions about convergence rates.
• Develop a general-purpose, robust, and efficient algorithm for identifying the best hyperparameter configuration in iterative machine learning settings.
• Enable early termination of poorly performing hyperparameter settings through principled online allocation of training resources.
• Demonstrate the practical superiority of the proposed method over standard baselines in terms of wall-clock time while maintaining comparable test accuracy.

Proposed method

• Formulate hyperparameter optimization as a non-stochastic best-arm identification problem, where each arm corresponds to a fixed hyperparameter configuration.
• Model the sequence of intermediate validation losses as non-stochastic, non-monotonic, and potentially non-smooth over time.
• Adopt the Successive Halving algorithm as the core method, which iteratively allocates more resources to promising arms based on performance at intermediate steps.
• Use a doubling trick to dynamically increase the budget per arm, resetting counters at each doubling to ensure interpretability and fairness.
• Apply the algorithm to iterative machine learning processes such as stochastic gradient descent, where intermediate model states can be evaluated without full convergence.
• Implement resource allocation strategies that prioritize arms showing early promise, reducing overall computation time while maintaining model quality.

Experimental results

Research questions

• RQ1 Can a non-stochastic best-arm identification framework be effectively applied to hyperparameter optimization in iterative machine learning?
• RQ2 How does the performance of Successive Halving compare to uniform allocation and Successive Rejects in terms of wall-clock time and convergence speed?
• RQ3 To what extent can early termination of poor hyperparameter configurations accelerate the optimization process without sacrificing final model accuracy?
• RQ4 How does the algorithm perform across different machine learning tasks with varying convergence dynamics and non-smooth loss trajectories?
• RQ5 What is the impact of computational cost differences between pulling an arm and observing its loss on the design and efficiency of the algorithm?

Key findings

• Successive Halving achieved the same low test error in kernel SVM tasks more than an order of magnitude faster than uniform allocation and Successive Rejects in terms of wall-clock time.
• On the ridge regression task, LUCB and lil’UCB achieved lower test error faster in terms of iterations, but were outperformed by Successive Halving and Successive Rejects in wall-clock time due to higher validation loss evaluation costs.
• In matrix completion with non-convex objectives, Successive Halving and Successive Rejects achieved target error rates two to eight times faster than uniform allocation in wall-clock time.
• The empirical results confirm that adaptive resource allocation based on intermediate performance leads to significant speedups, with consistent improvements across diverse datasets and models.
• The doubling trick with budget reset at each stage improved interpretability and performance, especially in settings with high variance in convergence behavior.
• The method remains effective even when loss sequences are non-monotonic and non-smooth, demonstrating robustness to real-world training dynamics.