[Paper Review] Automated Scalable Bayesian Inference via Hilbert Coresets

Trevor Campbell, Tamara Broderick|arXiv (Cornell University)|Oct 13, 2017

Gaussian Processes and Bayesian Inference52 references18 citations

TL;DR

This paper introduces Hilbert coresets—data compression techniques that enable fully automated, scalable Bayesian inference by constructing weighted subsets of data under a Hilbert space norm derived from log-likelihood functions. The method uses randomized projections to automate coreset construction and achieves high-quality posterior approximations with theoretical guarantees, reducing computational cost significantly across diverse models and datasets.

ABSTRACT

The automation of posterior inference in Bayesian data analysis has enabled experts and nonexperts alike to use more sophisticated models, engage in faster exploratory modeling and analysis, and ensure experimental reproducibility. However, standard automated posterior inference algorithms are not tractable at the scale of massive modern datasets, and modifications to make them so are typically model-specific, require expert tuning, and can break theoretical guarantees on inferential quality. Building on the Bayesian coresets framework, this work instead takes advantage of data redundancy to shrink the dataset itself as a preprocessing step, providing fully-automated, scalable Bayesian inference with theoretical guarantees. We begin with an intuitive reformulation of Bayesian coreset construction as sparse vector sum approximation, and demonstrate that its automation and performance-based shortcomings arise from the use of the supremum norm. To address these shortcomings we develop Hilbert coresets, i.e., Bayesian coresets constructed under a norm induced by an inner-product on the log-likelihood function space. We propose two Hilbert coreset construction algorithms---one based on importance sampling, and one based on the Frank-Wolfe algorithm---along with theoretical guarantees on approximation quality as a function of coreset size. Since the exact computation of the proposed inner-products is model-specific, we automate the construction with a random finite-dimensional projection of the log-likelihood functions. The resulting automated coreset construction algorithm is simple to implement, and experiments on a variety of models with real and synthetic datasets show that it provides high-quality posterior approximations and a significant reduction in the computational cost of inference.

Motivation & Objective

• To address the scalability limitations of automated Bayesian inference in large-scale datasets by reducing data size while preserving inferential quality.
• To overcome the shortcomings of existing coreset methods that rely on the supremum norm, which leads to poor performance and lack of automation.
• To develop a general-purpose, model-agnostic coreset construction framework that maintains theoretical approximation guarantees.
• To enable full automation of coreset construction without expert tuning, making it applicable to a broad class of Bayesian models.
• To demonstrate significant computational savings with high-quality posterior approximations across real and synthetic datasets.

Proposed method

• Reformulates Bayesian coreset construction as sparse vector sum approximation in a Hilbert space of log-likelihood functions.
• Introduces Hilbert coresets using a norm induced by an inner product on the log-likelihood function space, replacing the supremum norm to improve approximation quality.
• Proposes two coreset construction algorithms: one based on importance sampling and another using the Frank-Wolfe algorithm, both with theoretical approximation error bounds.
• Automates the method via random finite-dimensional projections of log-likelihood functions, avoiding model-specific inner-product computations.
• Uses weighted coreset approximation to replace the full dataset in any standard automated inference algorithm, enabling scalable posterior inference.
• Employs concentration inequalities to derive probabilistic bounds on coreset approximation error as a function of coreset size.

Experimental results

Research questions

• RQ1 Can Bayesian coreset construction be automated without model-specific tuning while maintaining theoretical approximation guarantees?
• RQ2 Does replacing the supremum norm with a Hilbert space norm on log-likelihood functions lead to better coreset approximation quality?
• RQ3 Can randomized projections of log-likelihood functions enable scalable and automated coreset construction across diverse Bayesian models?
• RQ4 How do Hilbert coreset methods compare to uniform subsampling and standard coresets in terms of posterior approximation accuracy and computational efficiency?
• RQ5 What is the theoretical relationship between coreset size and approximation error in the proposed Hilbert coreset framework?

Key findings

• The Hilbert coreset method achieves high-quality posterior approximations across logistic regression, Poisson regression, and directional clustering models on both real and synthetic datasets.
• Frank–Wolfe-based Hilbert coreset constructions outperform subsampling techniques, including uniform random subsampling and uniform coresets, in all evaluated models and datasets.
• Hilbert importance sampling coreset construction performs comparably to or better than uniform subsampling and uniform coresets, with consistent improvements in posterior approximation quality.
• The 2-norm and Fisher information norm yield similar performance in both importance sampling and Frank–Wolfe coreset constructions, validating the robustness of the norm choice.
• Theoretical bounds show that approximation error decreases with coreset size, with high probability bounds derived using concentration inequalities.
• The method reduces computational cost of inference by orders of magnitude while maintaining inferential quality, enabling scalable Bayesian inference on massive datasets.