[Paper Review] Fast Bayesian Inference with Batch Bayesian Quadrature via Kernel Recombination

Masaki Adachi, Satoshi Hayakawa+3 more|arXiv (Cornell University)|Jun 9, 2022

Gaussian Processes and Bayesian Inference4 citations

TL;DR

This paper proposes Batch Adaptive Sparse Quadrature (BASQ), a fast, parallel Bayesian quadrature method that uses kernel recombination to enable efficient, high-sample-efficiency inference of both posterior distributions and model evidence. By reweighting and recombining kernel-based quadrature points via a linearised importance proposal, BASQ achieves exponential convergence rates and outperforms batch WSABI and Nested Sampling in real-world datasets including lithium-ion battery analytics.

ABSTRACT

Calculation of Bayesian posteriors and model evidences typically requires numerical integration. Bayesian quadrature (BQ), a surrogate-model-based approach to numerical integration, is capable of superb sample efficiency, but its lack of parallelisation has hindered its practical applications. In this work, we propose a parallelised (batch) BQ method, employing techniques from kernel quadrature, that possesses an empirically exponential convergence rate. Additionally, just as with Nested Sampling, our method permits simultaneous inference of both posteriors and model evidence. Samples from our BQ surrogate model are re-selected to give a sparse set of samples, via a kernel recombination algorithm, requiring negligible additional time to increase the batch size. Empirically, we find that our approach significantly outperforms the sampling efficiency of both state-of-the-art BQ techniques and Nested Sampling in various real-world datasets, including lithium-ion battery analytics.

Motivation & Objective

• To address the lack of parallelization in Bayesian quadrature (BQ), which limits its practical use despite high sample efficiency.
• To enable simultaneous inference of both posterior distributions and model evidence in a single computation, avoiding repeated costly evaluations.
• To overcome the inefficiency of existing batch BQ methods like batch WSABI, which rely on computationally heavy multi-start optimization and suffer from local peak clustering.
• To develop a scalable, sparse sampling strategy that maintains high accuracy with minimal computational overhead when increasing batch size.
• To achieve exponential convergence rates in numerical integration for Bayesian inference using kernel recombination and Nyström-based test functions.

Proposed method

• Proposes a kernel recombination framework (RCHQ) that constructs a linearised importance proposal distribution $ g(x) = (1-r)\pi(x) + rA(x) $, blending prior and GP variance to guide batch selection.
• Uses subsampling from the proposal $ g(x) $ to form an empirical measure, enabling efficient and robust selection of quadrature points.
• Applies the Nyström method to construct finite test functions from the subsampled points, enabling tractable worst-case error minimization.
• Employs a single linear programming (LP) solver iteration to recombine kernel weights and select $ n $ batch points that minimize worst-case integration error.
• Introduces a sparse recombination algorithm that allows dynamic batch size increases with negligible computational cost.
• Leverages the analytical tractability of Gaussian processes with squared exponential kernels and Gaussian priors to compute mean and variance of integrals exactly.

Experimental results

Research questions

• RQ1 Can kernel recombination enable a parallel Bayesian quadrature method that achieves exponential convergence rates in practice?
• RQ2 How does kernel recombination compare to local penalization and multi-start optimization in batch point selection for BQ?
• RQ3 Can a single inference pipeline simultaneously estimate both model evidence and posterior distributions with high efficiency?
• RQ4 To what extent does the proposed method outperform state-of-the-art BQ and Nested Sampling in real-world, high-dimensional datasets?
• RQ5 How scalable and robust is the kernel recombination approach under increasing dimensionality and noisy likelihood evaluations?

Key findings

• BASQ achieves an empirically exponential convergence rate in numerical integration, significantly outperforming both batch WSABI and Nested Sampling in wall-clock time and sample efficiency.
• The kernel recombination method selects sparser, better-distributed batch points than local penalization in batch WSABI, reducing integration error by avoiding clustering around high-variance peaks.
• The method enables dynamic batch size increases with negligible additional cost, thanks to a sparse recombination algorithm that reuses existing computations.
• On lithium-ion battery analytics data, BASQ demonstrated superior performance compared to state-of-the-art BQ and Nested Sampling, with faster convergence and lower error.
• The use of a linearised importance proposal distribution reduces dependency on early-stage uncertainty estimates, improving robustness and stability in batch selection.