[Paper Review] Towards Automatic Bayesian Optimization: A first step involving acquisition functions

Eduardo C. Garrido‐Merchán, Luis C. Jariego Pérez|arXiv (Cornell University)|Mar 21, 2020

Gaussian Processes and Bayesian Inference14 references4 citations

TL;DR

This paper proposes a framework for automatic Bayesian optimization by dynamically selecting acquisition functions (AFs) using heuristic ensembles, such as weighted and hedge-based combinations, to reduce reliance on manual AF tuning. The approach achieves superior performance across benchmark functions and real hyperparameter tuning tasks, with the weighted AF consistently outperforming individual AFs and demonstrating robustness in noisy, high-dimensional settings.

ABSTRACT

Bayesian Optimization is the state of the art technique for the optimization of black boxes, i.e., functions where we do not have access to their analytical expression nor its gradients, they are expensive to evaluate and its evaluation is noisy. The most popular application of bayesian optimization is the automatic hyperparameter tuning of machine learning algorithms, where we obtain the best configuration of machine learning algorithms by optimizing the estimation of the generalization error of these algorithms. Despite being applied with success, bayesian optimization methodologies also have hyperparameters that need to be configured such as the probabilistic surrogate model or the acquisition function used. A bad decision over the configuration of these hyperparameters implies obtaining bad quality results. Typically, these hyperparameters are tuned by making assumptions of the objective function that we want to evaluate but there are scenarios where we do not have any prior information about the objective function. In this paper, we propose a first attempt over automatic bayesian optimization by exploring several heuristics that automatically tune the acquisition function of bayesian optimization. We illustrate the effectiveness of these heurisitcs in a set of benchmark problems and a hyperparameter tuning problem of a machine learning algorithm.

Motivation & Objective

• To reduce dependency on manual hyperparameter tuning in Bayesian optimization by automating acquisition function selection.
• To explore heuristic-based ensemble methods that combine multiple acquisition functions to improve optimization robustness.
• To evaluate whether automatic AF selection outperforms fixed, manually chosen acquisition functions in black-box optimization.
• To assess the effectiveness of ensemble AFs across diverse benchmark functions and real-world hyperparameter tuning tasks.
• To lay the foundation for fully automatic Bayesian optimization by identifying reliable, adaptive AF selection strategies.

Proposed method

• The authors propose a heuristic ensemble approach that combines multiple acquisition functions (e.g., Expected Improvement, Probability of Improvement, Lower Confidence Bound) using weighted and hedge-based strategies.
• Each acquisition function is evaluated iteratively, and their performance is dynamically weighted based on historical success in reducing regret.
• The method uses a grid search to optimize the acquisition function at each iteration, with a focus on minimizing absolute regret.
• A probabilistic graphical model is considered for future work, with potential use of deep learning to predict optimal AFs from GP states.
• The framework is tested on standard benchmark functions (Hartmann, Rastrigin) and a real hyperparameter tuning task on the Digits dataset using a Gradient Boosting classifier.
• The authors advocate for using slice sampling over maximum likelihood to avoid overfitting in low-evaluation regimes.

Experimental results

Research questions

• RQ1 Can heuristic ensembles of acquisition functions outperform individual, manually selected acquisition functions in Bayesian optimization?
• RQ2 How does dynamic weighting of acquisition functions affect optimization performance across different types of black-box functions?
• RQ3 Does automatic acquisition function selection reduce the need for expert tuning in hyperparameter optimization?
• RQ4 How do ensemble strategies perform on functions with multiple local optima, such as Rastrigin?
• RQ5 Can the proposed method generalize to real-world machine learning hyperparameter tuning tasks?

Key findings

• The weighted acquisition function consistently achieved the best performance across all benchmark functions, including Hartmann and Rastrigin, and tied for the best result in the hyperparameter tuning experiment.
• In the Hartmann function, pure exploitation AFs (EI and PI) outperformed exploration-based strategies, confirming that exploration is detrimental in unimodal, smooth functions.
• The ensemble AFs (weighted and hedge) demonstrated robustness by avoiding the worst-case performance, even when individual AFs failed.
• The Rastrigin function showed high standard deviations due to its many local optima and the use of grid search for AF optimization, suggesting a need for more sophisticated optimization of the acquisition function.
• The real hyperparameter tuning experiment on the Digits dataset confirmed that the weighted AF outperformed other strategies, with random search performing surprisingly well due to favorable parameter combinations.
• The study concludes that ensemble-based AF selection is a viable first step toward fully automatic Bayesian optimization, especially in scenarios with limited prior knowledge.