[Paper Review] Multi-fidelity Bayesian Optimisation with Continuous Approximations

Kirthevasan Kandasamy, Gautam Dasarathy+2 more|arXiv (Cornell University)|Mar 18, 2017

Machine Learning and Data Classification29 references95 citations

TL;DR

This paper introduces BOCA, a Bayesian optimisation method for continuous fidelity spaces, and provides theoretical regret guarantees showing it leverages continuous low-fidelity information to accelerate finding the optimum at high fidelity.

ABSTRACT

Bandit methods for black-box optimisation, such as Bayesian optimisation, are used in a variety of applications including hyper-parameter tuning and experiment design. Recently, \emph{multi-fidelity} methods have garnered considerable attention since function evaluations have become increasingly expensive in such applications. Multi-fidelity methods use cheap approximations to the function of interest to speed up the overall optimisation process. However, most multi-fidelity methods assume only a finite number of approximations. In many practical applications however, a continuous spectrum of approximations might be available. For instance, when tuning an expensive neural network, one might choose to approximate the cross validation performance using less data $N$ and/or few training iterations $T$. Here, the approximations are best viewed as arising out of a continuous two dimensional space $(N,T)$. In this work, we develop a Bayesian optimisation method, BOCA, for this setting. We characterise its theoretical properties and show that it achieves better regret than than strategies which ignore the approximations. BOCA outperforms several other baselines in synthetic and real experiments.

Motivation & Objective

• Motivate and formalize optimisation when a continuous spectrum of cheap approximations is available for expensive objectives.
• Develop a GP-based Bayesian optimisation algorithm that exploits continuous fidelities to improve sample efficiency.
• Provide theoretical regret bounds demonstrating gains over single-fidelity GP-UCB under smooth fidelity mappings.
• Empirically compare BOCA against baselines on synthetic and real-world problems to validate effectiveness.

Proposed method

• Model the objective as g(z, x) with f(x)=g(z bullet, x) where z lies in a continuous fidelity space Z.
• Use a Gaussian process prior on g with a product kernel k([z,x],[z',x'])=k0 * phi_Z(||z-z'||) * phi_X(||x-x'||).
• Construct a GP-UCB style upper confidence bound varphi_t(x)=mu_{t-1}(x)+sqrt(beta_t)*sigma_{t-1}(x) to select x_t.
• Define a fidelity subset Z_t(x_t) by three conditions that balance cost, information gap xi(z), and posterior uncertainty, and pick the cheapest admissible z_t (or z_bullet if none).
• Provide theoretical results (informal) showing BOCA achieves improved simple regret bounds relative to GP-UCB when g is smooth across Z, and degrade gracefully when smoothness is lower.

Experimental results

Research questions

• RQ1 Can continuous fidelity spaces be effectively leveraged in Bayesian optimisation to accelerate finding the high-fidelity optimum?
• RQ2 How should one select both the decision variable x and the fidelity z to balance information gain and cost in a continuum of fidelities?
• RQ3 What are the regret guarantees for BOCA, and how do they compare to single-fidelity GP-UCB under varying smoothness in the fidelity space?
• RQ4 Do empirical results on synthetic and real tasks support theoretical advantages of BOCA over finite-fidelity and non-multi-fidelity baselines?

Key findings

• BOCA outperforms baselines including GP-UCB, EI, MF-GP-UCB, and MF-SKO in synthetic experiments.
• The method adapts to the smoothness of g over the fidelity space, performing better when fidelities are highly informative about the high-fidelity objective.
• BOCA provides a theoretical regret bound indicating improved performance when the fidelity information gap xi is small (large kernel bandwidth in Z).
• The analysis shows most high-fidelity queries concentrate in a small region around the optimum, aided by shared information across fidelities.
• BOCA is extendable to discrete fidelity sets and other kernels, with scalability limitations acknowledged for high-dimensional settings.