[Paper Review] High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces

David Eriksson, Martin Jankowiak|arXiv (Cornell University)|Feb 27, 2021

Advanced Multi-Objective Optimization Algorithms30 references41 citations

TL;DR

SAASBO introduces a sparsity-inducing prior over GP length-scales to identify sparse axis-aligned subspaces, enabling sample-efficient Bayesian optimization in hundreds of dimensions using NUTS for inference. It outperforms baselines on synthetic and real-world problems without problem-specific hyperparameters.

ABSTRACT

Bayesian optimization (BO) is a powerful paradigm for efficient optimization of black-box objective functions. High-dimensional BO presents a particular challenge, in part because the curse of dimensionality makes it difficult to define -- as well as do inference over -- a suitable class of surrogate models. We argue that Gaussian process surrogate models defined on sparse axis-aligned subspaces offer an attractive compromise between flexibility and parsimony. We demonstrate that our approach, which relies on Hamiltonian Monte Carlo for inference, can rapidly identify sparse subspaces relevant to modeling the unknown objective function, enabling sample-efficient high-dimensional BO. In an extensive suite of experiments comparing to existing methods for high-dimensional BO we demonstrate that our algorithm, Sparse Axis-Aligned Subspace BO (SAASBO), achieves excellent performance on several synthetic and real-world problems without the need to set problem-specific hyperparameters.

Motivation & Objective

• Motivate the need for efficient Bayesian optimization in high-dimensional settings with limited evaluations.
• Propose a sparsity-inducing prior (SAAS) on GP kernel length-scales to identify relevant subspaces.
• Develop an inference framework (NUTS and MAP) to fit the SAAS-GP surrogate.
• Use EI to guide acquisition and demonstrate robustness and scalability across datasets.
• Show that SAASBO achieves strong performance without problem-specific hyperparameters.

Proposed method

• Define a Gaussian process surrogate with a SAAS prior that induces sparsity in inverse squared length scales to prefer axis-aligned subspaces.
• Use Half-Cauchy priors on shrinkage parameter and length-scales to adaptively turn dimensions on/off as data accumulate.
• Perform inference with No-U-Turn Sampler (NUTS) to sample kernel hyperparameters, or MAP as a faster alternative.
• Compute acquisition via Expected Improvement averaged over posterior kernel samples to handle hyperparameter uncertainty.
• Optimize EI (averaged over samples) with gradient-based methods to select the next query point in the full input space.

Experimental results

Research questions

• RQ1 Can SAAS priors identify a sparse, axis-aligned subspace that captures most of the function variation in high dimensions?
• RQ2 Does integrating over kernel hyperparameters via NUTS improve acquisition decisions and optimization performance compared to MAP?
• RQ3 How does SAASBO perform relative to state-of-the-art high-dimensional BO baselines on synthetic and real-world problems?
• RQ4 Is SAASBO robust to hyperparameter settings and scalable to problems with hundreds of dimensions?

Key findings

• SAASBO reliably identifies relevant subspaces and achieves superior model fit in high dimensions compared to MLE or weak-prior GPs.
• NUTS-based inference outperforms MAP in identifying the true subspace and yields better optimization performance.
• SAASBO demonstrates strong empirical performance across synthetic benchmarks and real-world problems with up to hundreds of dimensions without problem-specific hyperparameters.
• The effective subspace dimension grows adaptively during optimization, enabling increasingly flexible modeling as data accumulate.
• SAAS priors provide adaptive parsimony, turning off irrelevant dimensions while allowing important ones to contribute as evidence accumulates.