[Paper Review] Inducing Point Allocation for Sparse Gaussian Processes in High-Throughput Bayesian Optimisation

Henry B. Moss, Sebastian W. Ober+1 more|arXiv (Cornell University)|Jan 24, 2023

Gaussian Processes and Bayesian Inference4 citations

TL;DR

This paper proposes BO-specific inducing point allocation (IPA) for sparse Gaussian processes using quality-diversity decomposed determinantal point processes (DPPs), which prioritize high-fidelity modeling in promising regions of the search space. The method significantly improves optimization performance over existing IPA strategies by focusing modeling capacity on high-value areas, outperforming baselines in both synthetic and real-world high-throughput Bayesian optimization tasks.

ABSTRACT

Sparse Gaussian Processes are a key component of high-throughput Bayesian Optimisation (BO) loops; however, we show that existing methods for allocating their inducing points severely hamper optimisation performance. By exploiting the quality-diversity decomposition of Determinantal Point Processes, we propose the first inducing point allocation strategy designed specifically for use in BO. Unlike existing methods which seek only to reduce global uncertainty in the objective function, our approach provides the local high-fidelity modelling of promising regions required for precise optimisation. More generally, we demonstrate that our proposed framework provides a flexible way to allocate modelling capacity in sparse models and so is suitable broad range of downstream sequential decision making tasks.

Motivation & Objective

• To address the poor performance of existing inducing point allocation (IPA) strategies in high-throughput Bayesian optimization (BO), which prioritize global regression accuracy over local fidelity in promising regions.
• To develop a new IPA framework tailored specifically for BO, leveraging the quality-diversity decomposition of determinantal point processes (DPPs) to balance exploration and exploitation.
• To demonstrate that BO-specific IPA can dramatically improve optimization performance by focusing modeling capacity on regions of high expected improvement.
• To provide a general, flexible framework for allocating modeling resources in sparse models that can be applied to a broad range of sequential decision-making tasks.

Proposed method

• The method uses a quality-diversity decomposition of DPPs, where the quality function guides allocation toward high-value regions (e.g., promising optima), and diversity ensures coverage of the search space.
• It introduces a novel quality function based on the hyper-volume (HV) dominated by candidate points, referred to as HV-DPP, to guide inducing point placement in multi-objective BO.
• For single-objective BO, the method uses a quality function based on expected improvement (EI), adapted into a DPP framework to promote high-fidelity modeling in regions of high uncertainty reduction.
• The approach is compatible with stochastic variational Gaussian processes (SVGP), enabling scalable, high-throughput optimization with only 100 inducing points.
• The framework allows separate IPA for multiple models (e.g., objectives and constraints in multi-objective BO), using shared quality functions to coordinate allocation across models.
• Unlike standard SVGP, the method does not require Euclidean input spaces, enabling application to discrete structures like molecules and genes.

Figure 1: A toy problem showing two sparse GP surrogate models, one with its inducing points chosen using an existing method (top) and the other using one of our proposed BO-specific methods (bottom) that focuses modelling resources into promising areas of the search space. Our model is better suite

Experimental results

Research questions

• RQ1 Can existing inducing point allocation strategies for sparse GPs effectively support high-precision Bayesian optimization in high-throughput settings?
• RQ2 How can inducing point allocation be restructured to prioritize local high-fidelity modeling in promising regions rather than global regression accuracy?
• RQ3 Can quality-diversity decomposed DPPs be used to create a flexible, BO-optimized IPA framework that improves optimization performance?
• RQ4 To what extent can a sparse GP with only 100 inducing points match the performance of an exact GP in high-throughput multi-objective optimization tasks?

Key findings

• The proposed HV-DPP IPA strategy matches and exceeds the performance of exact Gaussian processes on the 4D ZDT3 multi-objective benchmark, outperforming all baselines in hyper-volume dominated.
• On the real-world automotive heat exchanger design task, an SVGP with only 100 inducing points using the proposed IPA achieves a Pareto front comparable to that of an expensive exact GP.
• The method reduces optimization overheads by an order of magnitude compared to exact GPs, while maintaining high optimization performance.
• The IMP-DPP variant, derived from the quality-diversity DPP framework, outperforms the conditional variance reduction IPA of Burt et al. (2019), which previously underperformed in BO settings.
• The framework enables effective high-throughput BO on discrete structures such as molecules and genes, extending beyond Euclidean input spaces.
• The quality function based on hyper-volume (HV) provides strong bias toward central regions of the Pareto front and is fast to evaluate, making it suitable for large-scale optimization.

(a)