[Paper Review] Computationally Efficient High-Dimensional Bayesian Optimization via Variable Selection

Yihang Shen, Carl Kingsford|arXiv (Cornell University)|Sep 20, 2021

Gaussian Processes and Bayesian Inference38 references4 citations

TL;DR

This paper proposes VS-BO, a computationally efficient Bayesian optimization method for high-dimensional black-box functions by automatically identifying and optimizing over an axis-aligned subspace of important variables, eliminating the need for pre-specifying embedding dimensions. The method achieves superior performance on synthetic and real-world problems like MOPTA08 and rover trajectory optimization, with faster convergence and lower computational cost than existing approaches.

ABSTRACT

Bayesian Optimization (BO) is a method for globally optimizing black-box functions. While BO has been successfully applied to many scenarios, developing effective BO algorithms that scale to functions with high-dimensional domains is still a challenge. Optimizing such functions by vanilla BO is extremely time-consuming. Alternative strategies for high-dimensional BO that are based on the idea of embedding the high-dimensional space to the one with low dimension are sensitive to the choice of the embedding dimension, which needs to be pre-specified. We develop a new computationally efficient high-dimensional BO method that exploits variable selection. Our method is able to automatically learn axis-aligned sub-spaces, i.e. spaces containing selected variables, without the demand of any pre-specified hyperparameters. We theoretically analyze the computational complexity of our algorithm and derive the regret bound. We empirically show the efficacy of our method on several synthetic and real problems.

Motivation & Objective

• To address the challenge of scaling Bayesian optimization to high-dimensional black-box functions where vanilla BO is computationally infeasible.
• To overcome the limitations of embedding-based BO methods that require pre-specifying the embedding dimension and risk projection errors.
• To develop a method that automatically identifies important variables without hyperparameter tuning, reducing computational complexity.
• To ensure the selected subspace is axis-aligned, avoiding issues from non-invertible projections in embedding methods.
• To theoretically analyze and empirically validate the computational efficiency and regret performance of the proposed method.

Proposed method

• The method assumes that only a subset of input variables significantly influence the objective function, partitioning inputs into important (x_ipt) and unimportant (x_nipt) variables.
• It employs a variable selection strategy based on Gaussian Process regression to iteratively identify important variables using acquisition function gradients.
• The algorithm performs Bayesian optimization only on the selected subspace of important variables, drastically reducing computational cost.
• A novel sampling strategy is introduced to prioritize variable selection, with a variant (VSBO mix) using a mixed strategy for comparison.
• The method avoids dimensionality reduction via projection by focusing on axis-aligned subspaces, ensuring all queries remain within the original domain.
• Theoretical analysis shows reduced computational complexity for both GP fitting and acquisition function optimization compared to vanilla BO.

Experimental results

Research questions

• RQ1 Can variable selection in high-dimensional Bayesian optimization eliminate the need for pre-specifying embedding dimensions?
• RQ2 How does automatic identification of important variables affect optimization efficiency and convergence speed?
• RQ3 Does axis-aligned sub-space optimization outperform embedding-based methods in terms of computational cost and solution quality?
• RQ4 What is the impact of variable selection accuracy on the regret and performance of the optimization algorithm?
• RQ5 How does VS-BO compare to state-of-the-art methods like SAASBO and Dragonfly under fixed iteration and time budgets?

Key findings

• VS-BO achieves faster convergence and lower computational cost than vanilla BO and embedding-based methods on high-dimensional synthetic functions like Branin, Hartmann6, and Styblinski-Tang4.
• On the 60-dimensional rover trajectory problem, VS-BO outperforms Dragonfly and other baselines under fixed iteration budgets, despite Dragonfly's strong performance under time constraints.
• On the 124-dimensional MOPTA08 problem, VS-BO achieves better optimization performance than baseline methods, though SAASBO outperforms it—though SAASBO is significantly slower.
• The variable selection process in VS-BO correctly identifies the most influential variables, with the top 5 selected variables showing significantly higher function value variance when perturbed.
• Empirical results show that the frequency of variable selection correlates strongly with variable importance, confirming the method's ability to learn relevant subspaces.
• The sampling strategy in VS-BO consistently outperforms the mixed strategy variant (VSBO mix), indicating the effectiveness of the proposed selection mechanism.