[Paper Review] Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimization

Benjamin Letham, Roberto Calandra+2 more|arXiv (Cornell University)|Jan 31, 2020

Advanced Multi-Objective Optimization Algorithms54 references32 citations

TL;DR

The paper analyzes why linear embeddings (random projections) for high-dimensional Bayesian optimization (HDBO) can underperform and introduces ALEBO, a method that uses a Mahalanobis kernel and constrained embeddings to improve modeling and the probability of containing an optimum.

ABSTRACT

Bayesian optimization (BO) is a popular approach to optimize expensive-to-evaluate black-box functions. A significant challenge in BO is to scale to high-dimensional parameter spaces while retaining sample efficiency. A solution considered in existing literature is to embed the high-dimensional space in a lower-dimensional manifold, often via a random linear embedding. In this paper, we identify several crucial issues and misconceptions about the use of linear embeddings for BO. We study the properties of linear embeddings from the literature and show that some of the design choices in current approaches adversely impact their performance. We show empirically that properly addressing these issues significantly improves the efficacy of linear embeddings for BO on a range of problems, including learning a gait policy for robot locomotion.

Motivation & Objective

• Identify why linear embeddings used in HDBO perform poorly and how to diagnose modeling issues.
• Develop improvements that yield better GP modelability when using linear embeddings.
• Propose ALEBO, a new linear-embedding HDBO method that outperforms existing approaches on benchmarks and real problems.

Proposed method

• Analyze properties of linear embeddings in BO and how clipping to ambient box bounds distorts the function.
• Introduce the Mahalanobis kernel tailored for linear embeddings to improve GP modeling.
• Constrain optimization in the embedding to a polytope defined by -1 <= B^†y <= 1 to avoid nonlinear clipping distortions.
• Estimate and maximize the probability that the embedding contains an optimum, P_opt, via Monte Carlo and linear programming.
• Propose ALEBO which uses hypersphere-based random projections, Mahalanobis kernel, and constrained acquisition optimization.
• Demonstrate ALEBO’s performance on synthetic HDBO benchmarks and real-world tasks (NAS, robot locomotion).

Experimental results

Research questions

• RQ1 How do linear embeddings distort function modeling in BO when box bounds are present?
• RQ2 Can a kernel tailored for linear embeddings (Mahalanobis kernel) improve GP modeling in the embedding?
• RQ3 How can we avoid nonlinear projection distortions due to clipping while maintaining a high chance of containing an optimum?
• RQ4 Does an adaptive linear-embedding BO method (ALEBO) outperform existing linear-embedding BO approaches on high-dimensional tasks?
• RQ5 What is the empirical performance of ALEBO on synthetic benchmarks and real-world optimization problems?

Key findings

• Linear embeddings can produce nonlinear distortions when projected points fall outside ambient bounds, harming GP modelability.
• A Mahalanobis kernel in the embedding, derived from the true subspace projection, improves GP modeling over ARD RBF kernels.
• Constraining y to satisfy -1 <= B^†y <= 1 avoids clipping distortions and enables effective linear modeling, albeit reducing the feasible search volume.
• The probability that the embedding contains an optimum, P_opt, increases with d_e > d and with hypersphere sampling, improving likelihood of success.
• ALEBO, combining hypersphere-based embedding, Mahalanobis kernel, and constrained optimization, outperforms REMBO and several HDBO baselines on benchmarks up to D=1000 and in real problems (NAS, robot locomotion).
• On constrained NAS tasks and robotic locomotion, ALEBO is among the best-performing methods among embedding-based approaches.