[Paper Review] Lower Bounds on the Worst-Case Complexity of Efficient Global Optimization

Wenjie Xu, Yuning Jiang+2 more|arXiv (Cornell University)|Sep 20, 2022

Advanced Optimization Algorithms Research4 citations

TL;DR

This paper establishes a unified lower bound on the worst-case sample complexity of efficient global optimization (EGO) for general kernel functions in the non-Bayesian, noiseless setting. By linking the complexity to the metric entropy of the reproducing kernel Hilbert space (RKHS), it proves that any deterministic algorithm requiring suboptimality gap $\epsilon$ must make $T = \Omega\left(\frac{\log \mathcal{N}(S(\mathcal{X}), 4\epsilon, \|\cdot\|_\infty)}{\log(R/\epsilon)}\right)$ queries, which nearly matches known upper bounds for squared exponential and Matérn kernels.

ABSTRACT

Efficient global optimization is a widely used method for optimizing expensive black-box functions such as tuning hyperparameter, and designing new material, etc. Despite its popularity, less attention has been paid to analyzing the inherent hardness of the problem although, given its extensive use, it is important to understand the fundamental limits of efficient global optimization algorithms. In this paper, we study the worst-case complexity of the efficient global optimization problem and, in contrast to existing kernel-specific results, we derive a unified lower bound for the complexity of efficient global optimization in terms of the metric entropy of a ball in its corresponding reproducing kernel Hilbert space~(RKHS). Specifically, we show that if there exists a deterministic algorithm that achieves suboptimality gap smaller than $ε$ for any function $f\in S$ in $T$ function evaluations, it is necessary that $T$ is at least $Ω\left(\frac{\log\mathcal{N}(S(\mathcal{X}), 4ε,\|\cdot\|_\infty)}{\log(\frac{R}ε)} ight)$, where $\mathcal{N}(\cdot,\cdot,\cdot)$ is the covering number, $S$ is the ball centered at $0$ with radius $R$ in the RKHS and $S(\mathcal{X})$ is the restriction of $S$ over the feasible set $\mathcal{X}$. Moreover, we show that this lower bound nearly matches the upper bound attained by non-adaptive search algorithms for the commonly used squared exponential kernel and the Matérn kernel with a large smoothness parameter $ν$, up to a replacement of $d/2$ by $d$ and a logarithmic term $\log\frac{R}ε$. That is to say, our lower bound is nearly optimal for these kernels.

Motivation & Objective

• To understand the fundamental limits of efficient global optimization (EGO) in terms of worst-case sample complexity.
• To close the gap between existing kernel-specific lower bounds and a general, kernel-agnostic complexity analysis.
• To establish a connection between the metric entropy of the RKHS and the inherent complexity of EGO problems.
• To analyze the noiseless, non-Bayesian setting to isolate the intrinsic complexity from statistical noise effects.
• To demonstrate that the derived lower bound nearly matches known upper bounds for popular kernels like SE and Matérn with large smoothness.

Proposed method

• Derives a general lower bound on the number of function evaluations $T$ required to achieve suboptimality gap $\epsilon$ for any function in a ball $S$ of radius $R$ in the RKHS.
• Uses the covering number $\mathcal{N}(S(\mathcal{X}), 4\epsilon, \|\cdot\|_\infty)$ as a measure of function space complexity.
• Applies information-theoretic arguments to show that $T$ must be at least $\Omega\left(\frac{\log \mathcal{N}(S(\mathcal{X}), 4\epsilon, \|\cdot\|_\infty)}{\log(R/\epsilon)}\right)$ for any deterministic algorithm.
• Establishes that this bound nearly matches the upper bounds achieved by non-adaptive search algorithms for the squared exponential and Matérn kernels.
• Uses results from functional analysis, including the equivalence of RKHS to Sobolev spaces for certain kernels, to derive lower bounds on covering numbers.
• Applies Stirling's approximation and Gaussian integral bounds to control error terms in the analysis of kernel matrix approximations.

Experimental results

Research questions

• RQ1 What is the fundamental worst-case sample complexity of efficient global optimization for general kernel functions?
• RQ2 How does the metric entropy of the RKHS ball relate to the query complexity of EGO algorithms?
• RQ3 Can a unified lower bound be derived that applies across different kernel classes, rather than being kernel-specific?
• RQ4 To what extent does the derived lower bound match existing upper bounds for popular kernels like squared exponential and Matérn?
• RQ5 Does the noiseless setting reveal the intrinsic complexity of EGO, independent of statistical noise?

Key findings

• The paper establishes a general lower bound on the worst-case sample complexity of EGO: $T = \Omega\left(\frac{\log \mathcal{N}(S(\mathcal{X}), 4\epsilon, \|\cdot\|_\infty)}{\log(R/\epsilon)}\right)$, where $\mathcal{N}$ is the covering number of the RKHS ball over the feasible set.
• For the squared exponential kernel, the derived lower bound nearly matches the known upper bound from non-adaptive algorithms, differing only by a factor of $d$ instead of $d/2$ and a logarithmic term.
• For the Matérn kernel with large smoothness parameter $\nu$, the lower bound matches the upper bound up to logarithmic factors, indicating near-optimality.
• The lower bound is derived in the noiseless, non-Bayesian setting, isolating the intrinsic complexity of the function class from statistical noise effects.
• The analysis shows that the complexity of EGO is fundamentally tied to the metric entropy of the RKHS, providing a general complexity measure independent of specific kernel structures.
• The result implies that existing algorithms for SE and Matérn kernels are nearly optimal in terms of worst-case sample complexity, and further improvements are unlikely without changing the problem structure.