[Paper Review] Practical Bayesian Optimization of Machine Learning Algorithms

Jasper Snoek, Hugo Larochelle+1 more|arXiv (Cornell University)|Jun 13, 2012

Gaussian Processes and Bayesian InferenceComputer Science23 references5,635 citations

TL;DR

Introduces fully Bayesian Bayesian optimization for hyperparameters using GP priors, with cost-aware and parallel acquisition, achieving expert-level or better tuning on diverse ML problems.

ABSTRACT

Machine learning algorithms frequently require careful tuning of model hyperparameters, regularization terms, and optimization parameters. Unfortunately, this tuning is often a "black art" that requires expert experience, unwritten rules of thumb, or sometimes brute-force search. Much more appealing is the idea of developing automatic approaches which can optimize the performance of a given learning algorithm to the task at hand. In this work, we consider the automatic tuning problem within the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). The tractable posterior distribution induced by the GP leads to efficient use of the information gathered by previous experiments, enabling optimal choices about what parameters to try next. Here we show how the effects of the Gaussian process prior and the associated inference procedure can have a large impact on the success or failure of Bayesian optimization. We show that thoughtful choices can lead to results that exceed expert-level performance in tuning machine learning algorithms. We also describe new algorithms that take into account the variable cost (duration) of learning experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization on a diverse set of contemporary algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks.

Motivation & Objective

• Automate the tuning of hyperparameters, regularization terms, and optimization settings for ML algorithms.
• Model generalization performance with a Gaussian process prior to guide efficient experimentation.
• Incorporate practical constraints such as varying trial duration and parallel evaluations into the optimization loop.

Proposed method

• Use a Gaussian process prior to model the unknown hyperparameter objective function f(x).
• Adopt acquisition functions like expected improvement (EI) and compare with GP-UCB.
• Fully Bayesian treatment of GP hyperparameters by marginalizing over them via Monte Carlo (EI with MCMC).
• Incorporate cost modeling by treating duration c(x) as a GP and optimizing EI per second.
• Enable parallel experiments by Monte Carlo averaging of acquisition under possible outcomes of pending evaluations.

Experimental results

Research questions

• RQ1 How does a fully Bayesian GP prior over hyperparameters affect Bayesian optimization performance?
• RQ2 Can cost-awareness (EI per second) and parallelism improve hyperparameter tuning efficiency in practice?
• RQ3 What is the impact of choosing different covariance functions (e.g., Matérn 5/2 vs squared exponential) on optimization success?
• RQ4 How does the integrated acquisition across pending evaluations influence next-point selection?
• RQ5 Do these methods outperform human experts on real ML problems?

Key findings

• Integrated over GP hyperparameters (GP EI MCMC) outperforms point-estimate hyperparameter strategies on benchmarks.
• EI per second accelerates wallclock-time efficiency by preferring faster-evaluating configurations.
• Parallelized GP EI MCMC (N x GP EI MCMC) can find better parameters faster than grid searches on large problems.
• Different covariance choices materially affect optimization success; Matérn 5/2 often yields more realistic function samples than squared exponential.
• On CIFAR-10, the GP EI MCMC approach achieved a 14.98% validation error versus expert settings.
• Across tasks (LDA, structured SVMs, CNNs), the proposed Bayesian optimization methods often surpass human expert performance and prior automatic methods.