[Paper Review] Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020

Ryan Turner, David Eriksson+5 more|arXiv (Cornell University)|Apr 20, 2021

Machine Learning and Data ClassificationComputer Science56 references163 citations

TL;DR

The paper reports that Bayesian optimization with surrogate models and ensembles significantly outperformed random search on ML hyperparameter tuning tasks in the NeurIPS 2020 Black-Box Optimization Challenge, with strong gains sustained on unseen problems.

ABSTRACT

This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning emphasis. It was based on tuning (validation set) performance of standard machine learning models on real datasets. This competition has widespread impact as black-box optimization (e.g., Bayesian optimization) is relevant for hyperparameter tuning in almost every machine learning project as well as many applications outside of machine learning. The final leaderboard was determined using the optimization performance on held-out (hidden) objective functions, where the optimizers ran without human intervention. Baselines were set using the default settings of several open-source black-box optimization packages as well as random search.

Motivation & Objective

• Motivate and evaluate derivative-free (black-box) optimizers for tuning ML model hyperparameters on real datasets.
• Provide a fair benchmarking protocol with hidden (held-out) objectives to compare optimizers.
• Assess baseline methods and the benefits of surrogate models and ensemble strategies.
• Offer practical insights on solver choices, open-source tooling, and challenges in ML hyperparameter optimization.

Proposed method

• Frame black-box optimization as min f(x) over a bounded domain with expensive evaluations.
• Use Bayesmark to normalize and score optimizer performance across heterogeneous problems.
• Provide a starter kit with default baselines (including random search) and open-source BO tools.
• Run a 16-iteration, batch-8 evaluation budget per problem to reflect wall-clock constraints.
• Encourage open-loop suggest-observe optimizers that can propose multiple points in parallel.
• Analyze results with bootstrap and randomized problem splits to assess ranking reliability.

Experimental results

Research questions

• RQ1 Is Bayesian optimization consistently superior to random search for ML hyperparameter tuning on real, diverse problems?
• RQ2 How do default BO packages compare to random search and to each other under practical budgets?
• RQ3 Do ensemble strategies among BO methods improve performance on heterogeneous ML tuning tasks?
• RQ4 What roles do open-source tools and warm-start/meta-learning play in achieving competitive results?
• RQ5 How robust are results to sampling variability and problem heterogeneity?

Key findings

• Most top-20 participants used surrogate-assisted optimization and achieved substantial gains over random search.
• 61 of 65 teams beat the random-search baseline on the final leaderboard; top-5 rankings were statistically robust under bootstrap analysis.
• Ensembles (combining multiple surrogates, acquisitions, or even optimizer families) yielded notable improvements over individual methods.
• Trust-region (TuRBO) and stochastic-RBF (pySOT) approaches were among the strongest baselines; ensembles that mix these with others often performed best.
• Open-source tooling and leveraging warm starts/meta-learning strategies contributed to practical performance gains; several top teams integrated established BO components rather than bespoke solutions.