[Paper Review] HEBO: Heteroscedastic Evolutionary Bayesian Optimisation.

Alexander I. Cowen-Rivers, Wenlong Lyu+5 more|arXiv (Cornell University)|Dec 7, 2020

Advanced Multi-Objective Optimization Algorithms24 references19 citations

TL;DR

HEBO proposes a heteroscedastic evolutionary Bayesian optimisation framework that addresses hyper-parameter tuning by modeling non-stationary, heteroscedastic objective functions through multi-objective acquisition ensembles and robust acquisition maximisation. The method achieves superior configuration queries and empirical performance over non-robust and conventional approaches.

ABSTRACT

Inspired by the increasing desire to efficiently tune machine learning hyper-parameters, in this work we rigorously analyse conventional and non-conventional assumptions inherent to Bayesian optimisation. Across an extensive set of experiments we conclude that: 1) the majority of hyper-parameter tuning tasks exhibit heteroscedasticity and non-stationarity, 2) multi-objective acquisition ensembles with Pareto-front solutions significantly improve queried configurations, and 3) robust acquisition maximisation affords empirical advantages relative to its non-robust counterparts. We hope these findings may serve as guiding principles, both for practitioners and for further research in the field.

Motivation & Objective

• To investigate the assumptions underlying conventional Bayesian optimisation in hyper-parameter tuning.
• To address the prevalence of heteroscedasticity and non-stationarity in real-world hyper-parameter optimisation tasks.
• To improve configuration query quality through Pareto-front-based multi-objective acquisition ensembles.
• To enhance robustness in acquisition maximisation for better empirical performance.
• To provide guiding principles for practitioners and future research in Bayesian optimisation.

Proposed method

• The method models heteroscedasticity in the objective function by learning uncertainty estimates that vary across the input space.
• It employs an evolutionary strategy to explore the search space and guide acquisition function optimisation.
• A multi-objective acquisition ensemble is used, combining multiple acquisition functions to promote diversity and convergence to the Pareto front.
• Robust acquisition maximisation is implemented to handle noisy or uncertain evaluations, improving stability and performance.
• The framework integrates evolutionary search with Bayesian optimisation, allowing flexible and adaptive exploration.
• The approach is evaluated across a broad set of hyper-parameter tuning tasks to validate its effectiveness.

Experimental results

Research questions

• RQ1 To what extent do real-world hyper-parameter tuning tasks exhibit heteroscedasticity and non-stationarity?
• RQ2 How does using a multi-objective acquisition ensemble with Pareto-front solutions affect the quality of queried configurations?
• RQ3 What is the empirical impact of robust acquisition maximisation compared to non-robust variants?
• RQ4 Can heteroscedastic modelling improve Bayesian optimisation performance in non-stationary settings?
• RQ5 What design principles emerge from analysing conventional and non-conventional assumptions in Bayesian optimisation?

Key findings

• The majority of hyper-parameter tuning tasks exhibit significant heteroscedasticity and non-stationarity, challenging standard Bayesian optimisation assumptions.
• Multi-objective acquisition ensembles with Pareto-front solutions lead to a measurable improvement in the quality of queried configurations.
• Robust acquisition maximisation consistently outperforms non-robust counterparts in terms of empirical performance.
• The proposed HEBO framework achieves better configuration search efficiency and convergence compared to baseline methods.
• These findings suggest that heteroscedastic and robust modelling should be prioritised in future Bayesian optimisation systems.
• The study provides actionable insights and guiding principles for practitioners and future research in hyper-parameter optimisation.