[Paper Review] The Behavior and Convergence of Local Bayesian Optimization

Kaiwen Wu, Kyurae Kim+2 more|arXiv (Cornell University)|May 24, 2023

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper provides the first rigorous theoretical analysis of local Bayesian optimization, focusing on the GIBO algorithm. It establishes convergence rates for both noiseless and noisy settings by bounding gradient uncertainty, showing that local BO achieves fast convergence with polynomial sample complexity, even in high dimensions.

ABSTRACT

A recent development in Bayesian optimization is the use of local optimization strategies, which can deliver strong empirical performance on high-dimensional problems compared to traditional global strategies. The "folk wisdom" in the literature is that the focus on local optimization sidesteps the curse of dimensionality; however, little is known concretely about the expected behavior or convergence of Bayesian local optimization routines. We first study the behavior of the local approach, and find that the statistics of individual local solutions of Gaussian process sample paths are surprisingly good compared to what we would expect to recover from global methods. We then present the first rigorous analysis of such a Bayesian local optimization algorithm recently proposed by Müller et al. (2021), and derive convergence rates in both the noisy and noiseless settings.

Motivation & Objective

• To formally analyze the behavior and convergence of local Bayesian optimization, which has shown strong empirical performance but lacks theoretical grounding.
• To address the gap in understanding why local BO performs well on high-dimensional problems despite the curse of dimensionality.
• To provide theoretical guarantees for a local BO algorithm (GIBO) under standard GP assumptions used in global BO.
• To quantify the impact of noise and batch size on convergence speed in local optimization.
• To evaluate the quality of individual local solutions found by local BO on typical GP sample paths.

Proposed method

• Uses the GIBO algorithm (Gaussian process-based iterative Bayesian optimization) with active gradient learning to focus on local regions.
• Derives upper bounds on the uncertainty of the gradient estimate using a novel analysis of the posterior variance in GP regression.
• Relates gradient uncertainty bounds to bias in the gradient estimate to derive convergence rates for both noiseless and noisy functions.
• Analyzes batch size selection by optimizing polynomial and logarithmic growth rates to minimize convergence error.
• Empirically evaluates the performance of a modified GIBO variant on GP sample paths across varying dimensions and noise levels.
• Uses theoretical bounds to assess the looseness of convergence guarantees by comparing upper bounds to empirical uncertainty reduction.

Figure 1 : The unreasonable effectiveness of locally optimizing GP sample paths. (Top row): Distributions of local solutions found when locally optimizing GP sample paths in various numbers of dimensions, with varying amounts of noise. (Bottom left): The minimum sample complexity of grid search requ

Experimental results

Research questions

• RQ1 How good are the local solutions found by local Bayesian optimization on typical high-dimensional GP sample paths?
• RQ2 Can we provide theoretical convergence guarantees for a local Bayesian optimization algorithm under standard GP assumptions?
• RQ3 What is the impact of noise on the convergence rate of local BO?
• RQ4 How does batch size selection affect the convergence speed of local BO?
• RQ5 How does the uncertainty in gradient estimation evolve during local optimization, and how tight are the theoretical bounds?

Key findings

• Local Bayesian optimization finds remarkably good solutions on high-dimensional GP sample paths, with median objective values equivalent to those of grid search requiring exponentially larger sample sizes.
• The paper establishes a convergence rate of $\mathcal{O}(n^{-1/3})$ for the noiseless case with optimal batch size $b_t = d t^2$, which is the fastest known rate under these settings.
• In the noisy setting, the convergence rate is $\mathcal{O}(\sigma d^{4/3} n^{-1/3} \log^2 n)$, showing that noise increases the sample complexity but still allows for fast convergence.
• Theoretical bounds on gradient uncertainty are shown to be loose in practice, as empirical uncertainty reduction significantly outperforms the upper bounds.
• With batch size $b_t = d+1$, the algorithm achieves convergence with $\min_t \|\nabla f(x_t)\|^2 \leq \frac{2L(f(x_1) - f^*)}{T}$, demonstrating fast convergence under ideal conditions.
• The ReLU function is successfully handled via subgradient estimation, with posterior mean gradients consistently lying within the subdifferential $[0,1]$.

(a) fixed dimension, varying batch sizes