[Paper Review] BOHB: Robust and Efficient Hyperparameter Optimization at Scale

Stefan Falkner, Aaron Klein+1 more|arXiv (Cornell University)|Jul 4, 2018

Advanced Neural Network Applications3 references180 citations

TL;DR

BOHB combines Bayesian optimization and Hyperband to achieve fast, robust hyperparameter optimization that scales to high dimensions and parallel resources, outperforming both BO and Hyperband across diverse tasks.

ABSTRACT

Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible. On the other hand, bandit-based configuration evaluation approaches based on random search lack guidance and do not converge to the best configurations as quickly. Here, we propose to combine the benefits of both Bayesian optimization and bandit-based methods, in order to achieve the best of both worlds: strong anytime performance and fast convergence to optimal configurations. We propose a new practical state-of-the-art hyperparameter optimization method, which consistently outperforms both Bayesian optimization and Hyperband on a wide range of problem types, including high-dimensional toy functions, support vector machines, feed-forward neural networks, Bayesian neural networks, deep reinforcement learning, and convolutional neural networks. Our method is robust and versatile, while at the same time being conceptually simple and easy to implement.

Motivation & Objective

• Motivate practical hyperparameter optimization (HPO) that balances fast initial results with strong final performance.
• Achieve scalability to high-dimensional, mixed discrete-continuous configuration spaces.
• Leverage parallel resources effectively while maintaining robustness across problem domains.
• Provide a simple, efficient, and flexible HPO method suitable for diverse ML tasks.

Proposed method

• Integrate Hyperband (HB) with a Bayesian optimization component based on Tree Parzen Estimators (TPE) to guide configuration search.
• Replace random configuration sampling in HB with model-based sampling using a single multidimensional KDE.
• Use budgets to evaluate configurations at varying fidelities and apply Successive Halving on a set of BO-sampled configurations.
• Maintain results across budgets to build progressively better models, prioritizing the largest budget for final decisions.
• Include a fraction of random configurations to preserve exploration and theoretical HB guarantees.
• Parallelize by sharing observations across SH runs and interleaving SH runs to utilize many workers efficiently.

Experimental results

Research questions

• RQ1 Can combining Bayesian optimization with Hyperband yield both strong anytime performance and strong final performance?
• RQ2 How can BO be adapted to work with multi-budget evaluations and interaction effects in high-dimensional spaces?
• RQ3 Does BOHB scale to high-dimensional, mixed categorical/continuous hyperparameter spaces and leverage parallel compute effectively?
• RQ4 How does BOHB perform across diverse ML tasks (SVMs, neural nets, Bayesian nets, RL, CNNs) compared to baselines?
• RQ5 What are the practical hyperparameter settings and robustness characteristics of BOHB across problems?

Key findings

• BOHB consistently outperforms both Bayesian optimization and Hyperband across a wide range of problem types.
• BOHB achieves fast initial progress and rapid convergence to near-global optima, outperforming HB in final performance and BO in early progress.
• BOHB scales to high-dimensional spaces and heterogeneous hyperparameter types (binary, categorical, integer, continuous).
• BOHB demonstrates strong performance gains in neural networks, Bayesian neural networks, reinforcement learning, and convolutional networks compared to baselines.
• BOHB enables effective parallelization with near-linear speedups up to a practical number of workers, using shared observational data across SH runs.