[Paper Review] An Empirical Study of Assumptions in Bayesian Optimisation

Alexander I. Cowen-Rivers, Wenlong Lyu+8 more|arXiv (Cornell University)|Dec 7, 2020

Gaussian Processes and Bayesian InferenceComputer Science84 references21 citations

TL;DR

This paper empirically investigates core assumptions in Bayesian optimization for machine learning hyper-parameter tuning. It demonstrates that real-world tasks often exhibit heteroscedasticity and non-stationarity, and finds that multi-objective acquisition ensembles with Pareto-front solutions and robust acquisition maximization significantly improve optimization performance over standard approaches.

ABSTRACT

Inspired by the increasing desire to efficiently tune machine learning hyper-parameters, in this work we rigorously analyse conventional and non-conventional assumptions inherent to Bayesian optimisation. Across an extensive set of experiments we conclude that: 1) the majority of hyper-parameter tuning tasks exhibit heteroscedasticity and non-stationarity, 2) multi-objective acquisition ensembles with Pareto-front solutions significantly improve queried configurations, and 3) robust acquisition maximisation affords empirical advantages relative to its non-robust counterparts. We hope these findings may serve as guiding principles, both for practitioners and for further research in the field.

Motivation & Objective

• To investigate the validity of conventional assumptions in Bayesian optimization, particularly stationarity and homoscedasticity, in real-world hyper-parameter tuning tasks.
• To evaluate the effectiveness of multi-objective acquisition functions that leverage Pareto-front solutions in improving configuration quality.
• To assess whether robust acquisition maximization—accounting for uncertainty in acquisition function optimization—leads to better empirical performance.
• To provide actionable insights and guiding principles for practitioners and future research in Bayesian optimization.

Proposed method

• Conducted extensive empirical experiments across a diverse set of hyper-parameter tuning tasks to evaluate model assumptions.
• Evaluated multi-objective acquisition ensembles that optimize for multiple criteria simultaneously, using Pareto-front solutions to balance trade-offs.
• Implemented robust acquisition maximization by incorporating uncertainty in the optimization of the acquisition function, reducing sensitivity to noisy evaluations.
• Compared robust and non-robust acquisition maximization strategies under identical experimental conditions to isolate their performance differences.
• Analyzed the statistical properties of objective functions, including heteroscedasticity and non-stationarity, across benchmark tasks.
• Used standard Bayesian optimization frameworks (e.g., Gaussian processes) as the base model, with modifications to acquisition function design and optimization.

Experimental results

Research questions

• RQ1 To what extent do real-world hyper-parameter tuning tasks violate the assumptions of stationarity and homoscedasticity in Bayesian optimization?
• RQ2 How do multi-objective acquisition ensembles based on Pareto-front solutions compare to single-objective acquisition functions in terms of configuration quality?
• RQ3 Does robust acquisition maximization—accounting for uncertainty in the acquisition function optimization—yield measurable performance improvements over non-robust counterparts?
• RQ4 Which assumptions in standard Bayesian optimization are most frequently violated in practice, and how do these violations affect optimization outcomes?

Key findings

• The majority of hyper-parameter tuning tasks exhibit significant heteroscedasticity and non-stationarity, challenging the standard assumption of stationary, homoscedastic objective functions.
• Multi-objective acquisition ensembles that incorporate Pareto-front solutions lead to a measurable improvement in the quality of queried configurations compared to single-objective methods.
• Robust acquisition maximization consistently outperforms non-robust counterparts, demonstrating empirical advantages in convergence and solution quality.
• The findings suggest that conventional Bayesian optimization assumptions are frequently violated in practice, indicating a need for more flexible and robust optimization strategies.