[Paper Review] Bayesian Coreset Construction via Greedy Iterative Geodesic Ascent

Trevor Campbell, Tamara Broderick|arXiv (Cornell University)|Feb 5, 2018

Gaussian Processes and Bayesian InferenceComputer Science26 references17 citations

TL;DR

This paper proposes Greedy Iterative Geodesic Ascent (GIGA), a novel Bayesian coreset construction algorithm that optimally scales the coreset log-likelihood to minimize posterior approximation error. By leveraging geodesic ascent on the manifold of log-likelihood vectors, GIGA achieves geometric decay in error with coreset size, reducing posterior uncertainty approximation error by orders of magnitude compared to prior methods while maintaining fast computation.

ABSTRACT

Coherent uncertainty quantification is a key strength of Bayesian methods. But modern algorithms for approximate Bayesian posterior inference often sacrifice accurate posterior uncertainty estimation in the pursuit of scalability. This work shows that previous Bayesian coreset construction algorithms---which build a small, weighted subset of the data that approximates the full dataset---are no exception. We demonstrate that these algorithms scale the coreset log-likelihood suboptimally, resulting in underestimated posterior uncertainty. To address this shortcoming, we develop greedy iterative geodesic ascent (GIGA), a novel algorithm for Bayesian coreset construction that scales the coreset log-likelihood optimally. GIGA provides geometric decay in posterior approximation error as a function of coreset size, and maintains the fast running time of its predecessors. The paper concludes with validation of GIGA on both synthetic and real datasets, demonstrating that it reduces posterior approximation error by orders of magnitude compared with previous coreset constructions.

Motivation & Objective

• To address the suboptimal scaling of coreset log-likelihood in existing Bayesian coreset methods, which leads to underestimated posterior variance.
• To develop a coreset construction algorithm that maintains theoretical guarantees on posterior approximation error while scaling efficiently.
• To achieve geometric decay in posterior approximation error as a function of coreset size, ensuring uniformly bounded relative error across models.
• To provide a method that outperforms prior coreset constructions in both synthetic and real-world Bayesian inference tasks.

Proposed method

• GIGA formulates coreset construction as a cardinality-constrained vector approximation problem in the space of log-likelihood functions.
• It uses greedy iterative geodesic ascent to iteratively select data points and update weights by maximizing the geodesic distance to the full-data log-likelihood vector.
• The algorithm operates on the manifold of log-likelihood vectors, ensuring optimal alignment with the full-data likelihood under a Riemannian geometry framework.
• At each step, GIGA computes the optimal weight update via constrained optimization on the unit sphere, preserving the geometric structure of the likelihood space.
• It maintains computational efficiency by avoiding full Hessian computation, relying instead on local geodesic approximations.
• The method ensures that the coreset log-likelihood scales optimally to match the full-data log-likelihood, preventing artificial data inflation.

Experimental results

Research questions

• RQ1 Can existing Bayesian coreset methods be shown to systematically underestimate posterior variance due to suboptimal log-likelihood scaling?
• RQ2 Is there a geometric framework that enables optimal coreset construction with provably better posterior approximation error?
• RQ3 Can a greedy algorithm on the likelihood manifold achieve exponential decay in posterior approximation error with increasing coreset size?
• RQ4 Does optimal log-likelihood scaling lead to uniformly bounded relative error across diverse statistical models?
• RQ5 How does GIGA compare empirically to prior coreset methods in terms of posterior accuracy and computational cost?

Key findings

• GIGA reduces posterior approximation error by orders of magnitude compared to previous coreset construction methods on both synthetic and real datasets.
• The algorithm achieves geometric (exponential) decay in posterior approximation error as a function of coreset size, as proven in Theorems 3.1 and 3.2.
• Unlike prior methods, GIGA maintains uniformly bounded relative error across all models, even in the limit of small coreset sizes.
• GIGA's optimal log-likelihood scaling prevents artificial data inflation, resulting in more accurate uncertainty quantification.
• Empirical validation on regression models and synthetic vector approximation problems confirms GIGA's superior performance in preserving posterior variance.
• The method maintains the same computational complexity as state-of-the-art coreset algorithms while significantly improving posterior accuracy.