[Paper Review] Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets

Aaron Klein, Stefan Falkner+3 more|arXiv (Cornell University)|May 23, 2016

Machine Learning and Data Classification317 citations

TL;DR

The paper introduces Fabolas, a Bayesian optimization method that models loss and cost as a function of dataset size, enabling hyperparameter optimization on large datasets by evaluating cheaper subsamples and extrapolating to the full data.

ABSTRACT

Bayesian optimization has become a successful tool for hyperparameter optimization of machine learning algorithms, such as support vector machines or deep neural networks. Despite its success, for large datasets, training and validating a single configuration often takes hours, days, or even weeks, which limits the achievable performance. To accelerate hyperparameter optimization, we propose a generative model for the validation error as a function of training set size, which is learned during the optimization process and allows exploration of preliminary configurations on small subsets, by extrapolating to the full dataset. We construct a Bayesian optimization procedure, dubbed Fabolas, which models loss and training time as a function of dataset size and automatically trades off high information gain about the global optimum against computational cost. Experiments optimizing support vector machines and deep neural networks show that Fabolas often finds high-quality solutions 10 to 100 times faster than other state-of-the-art Bayesian optimization methods or the recently proposed bandit strategy Hyperband.

Motivation & Objective

• Motivate hyperparameter optimization for large datasets where full evaluations are costly or impractical.
• Propose a principled way to use subsampled datasets as an environmental variable to accelerate search.
• Develop a Bayesian optimization framework that extrapolates performance to the full dataset from smaller subsets.
• Enable automatic trade-offs between information gain and computational cost while targeting the full data performance.

Proposed method

• Model loss and cost as functions of hyperparameters and dataset size via Gaussian processes with a customized kernel.
• Extend the kernel with a finite-rank component in dataset size to allow extrapolation from s in [0,1] to s=1.
• Use Entropy Search as the acquisition function to maximize information gain about the full-data optimum per unit time.
• Incorporate overhead time into the acquisition to reflect wall-clock cost rather than just evaluation cost.
• Initialize with a biased design toward cheap evaluations to learn scaling with dataset size.
• Provide an open-source implementation (RoBO) for reproducibility.

Experimental results

Research questions

• RQ1 Can subsample-evaluations be used to reliably infer hyperparameter performance on the full dataset?
• RQ2 How should one model loss and computational cost as a function of dataset size to enable extrapolation to the full data?
• RQ3 Does Fabolas outperform standard Bayesian optimization, MTBO, and Hyperband in discovering high-quality hyperparameters for large datasets?
• RQ4 What is the impact of including evaluation overhead in the decision rule for selecting (x, s)?

Key findings

• Fabolas often finds high-quality hyperparameters 10 to 100 times faster than other Bayesian optimization methods or Hyperband.
• On SVM and deep neural network tasks, Fabolas achieves substantial wall-clock speedups while matching or surpassing full-data baselines.
• Using a continuous dataset size variable allows learning correlations without ever evaluating at full size in many cases.
• Compared to MTBO, Hyperband, and standard BO, Fabolas provides faster convergence to good incumbents across multiple datasets.
• The method remains effective across CNNs and residual networks, though speedups vary with scaling properties of the model and data.