[Paper Review] Batch Bayesian Optimization via Local Penalization

Javier González, Zhenwen Dai+2 more|arXiv (Cornell University)|May 29, 2015

Gaussian Processes and Bayesian Inference218 citations

TL;DR

This paper proposes Batch Bayesian Optimization via Local Penalization (BBP-LP), a computationally efficient heuristic that uses an estimated Lipschitz constant to enforce local repulsion among batch points, avoiding costly joint modeling of batch interactions. The method achieves performance comparable to complex batch BO approaches with minimal overhead, significantly outperforming sequential and other batch methods in terms of convergence speed and wall-clock time on benchmark functions and real-world experiments.

ABSTRACT

The popularity of Bayesian optimization methods for efficient exploration of parameter spaces has lead to a series of papers applying Gaussian processes as surrogates in the optimization of functions. However, most proposed approaches only allow the exploration of the parameter space to occur sequentially. Often, it is desirable to simultaneously propose batches of parameter values to explore. This is particularly the case when large parallel processing facilities are available. These facilities could be computational or physical facets of the process being optimized. E.g. in biological experiments many experimental set ups allow several samples to be simultaneously processed. Batch methods, however, require modeling of the interaction between the evaluations in the batch, which can be expensive in complex scenarios. We investigate a simple heuristic based on an estimate of the Lipschitz constant that captures the most important aspect of this interaction (i.e. local repulsion) at negligible computational overhead. The resulting algorithm compares well, in running time, with much more elaborate alternatives. The approach assumes that the function of interest, $f$, is a Lipschitz continuous function. A wrap-loop around the acquisition function is used to collect batches of points of certain size minimizing the non-parallelizable computational effort. The speed-up of our method with respect to previous approaches is significant in a set of computationally expensive experiments.

Motivation & Objective

• To address the computational bottleneck in batch Bayesian optimization, where modeling interactions between batch points is expensive.
• To enable efficient parallel evaluation of expensive black-box functions by designing batches that avoid redundant or clustered evaluations.
• To develop a heuristic that captures essential interaction effects—local repulsion—using minimal computational overhead.
• To improve convergence speed and wall-clock efficiency compared to existing batch and sequential Bayesian optimization methods.
• To provide a scalable, acquisition-function-agnostic method applicable to real-world problems with costly function evaluations.

Proposed method

• The method introduces a penalized acquisition function that incorporates local repulsion between batch points using an estimated Lipschitz constant of the objective function.
• It models the interaction between batch points not through full joint GP inference, but via a heuristic penalty that discourages points from being too close in input space.
• The penalty is derived from the function's L-Lipschitz continuity, ensuring that nearby points in the batch do not yield similar function values.
• The approach avoids recomputing the GP posterior after each point is added, reducing computational cost compared to full batch optimization.
• It is compatible with any acquisition function (e.g., Expected Improvement, Upper Confidence Bound), making it broadly applicable.
• The method naturally suggests a Latin hypercube design for initial batch selection when no data is available.

Experimental results

Research questions

• RQ1 Can a simple heuristic based on Lipschitz continuity effectively model batch interactions in Bayesian optimization without full joint modeling?
• RQ2 How does the proposed local penalization method compare in performance and efficiency to more complex batch Bayesian optimization approaches?
• RQ3 Does the method maintain or improve convergence speed and final solution quality compared to sequential Bayesian optimization under parallel evaluation constraints?
• RQ4 Can the method be efficiently applied across diverse problem types, including synthetic functions and real-world machine learning and wet-lab experiments?
• RQ5 To what extent does the method reduce computational overhead while preserving the quality of batch selection?

Key findings

• On the Cosines function, BBP-LP (LP-EI and LP-UCB) significantly outperformed other batch and sequential methods in terms of convergence speed and wall-clock time after 10 batches.
• In the wet-lab experiment, where function evaluations were more expensive, LP-UCB achieved the best performance, demonstrating superior efficiency under high evaluation cost.
• For the SVR hyperparameter tuning task, all batch methods outperformed sequential policies due to parallelization, but BBP-LP maintained strong performance with minimal overhead.
• The method achieved comparable or better results than complex batch methods while incurring negligible computational cost, making it highly efficient for large-scale or high-cost problems.
• The use of a Lipschitz-based penalty effectively captured local repulsion, enabling better exploration than random or non-penalized batch strategies.
• The method's performance was robust across different acquisition functions (EI and UCB), confirming its generality and scalability.