[Paper Review] Sparse Variational Inference: Bayesian Coresets from Scratch

Trevor Campbell, Boyan Beronov|arXiv (Cornell University)|Jun 7, 2019

Gaussian Processes and Bayesian Inference59 references4 citations

TL;DR

This paper proposes a fully automated sparse variational inference framework for Bayesian coreset construction by formulating coreset selection as sparsity-constrained variational inference in an exponential family. The method uses greedy optimization on the Riemannian manifold of the coreset posterior, achieving state-of-the-art posterior approximation with orders-of-magnitude lower KL divergence than prior methods, which are limited by fixed tangent plane approximations.

ABSTRACT

The proliferation of automated inference algorithms in Bayesian statistics has provided practitioners newfound access to fast, reproducible data analysis and powerful statistical models. Designing automated methods that are also both computationally scalable and theoretically sound, however, remains a significant challenge. Recent work on Bayesian coresets takes the approach of compressing the dataset before running a standard inference algorithm, providing both scalability and guarantees on posterior approximation error. But the automation of past coreset methods is limited because they depend on the availability of a reasonable coarse posterior approximation, which is difficult to specify in practice. In the present work we remove this requirement by formulating coreset construction as sparsity-constrained variational inference within an exponential family. This perspective leads to a novel construction via greedy optimization, and also provides a unifying information-geometric view of present and past methods. The proposed Riemannian coreset construction algorithm is fully automated, requiring no problem-specific inputs aside from the probabilistic model and dataset. In addition to being significantly easier to use than past methods, experiments demonstrate that past coreset constructions are fundamentally limited by the fixed coarse posterior approximation; in contrast, the proposed algorithm is able to continually improve the coreset, providing state-of-the-art Bayesian dataset summarization with orders-of-magnitude reduction in KL divergence to the exact posterior.

Motivation & Objective

• To address the lack of automation in existing Bayesian coreset methods, which rely on user-specified coarse posterior approximations.
• To eliminate the need for problem-specific inputs in coreset construction beyond the model and dataset.
• To develop a theoretically grounded, scalable coreset method with provable posterior approximation guarantees.
• To unify and extend prior coreset methods through a Riemannian information-geometric perspective.
• To demonstrate that continual improvement along the coreset manifold leads to superior posterior approximation compared to fixed-tangent methods.

Proposed method

• Formulates coreset construction as sparsity-constrained variational inference within an exponential family, enabling tractable KL divergence gradient estimation.
• Employs a greedy optimization algorithm on the Riemannian manifold of the coreset posterior to iteratively improve the coreset.
• Uses unbiased gradient estimation of the KL divergence between the coreset posterior and the true posterior to guide selection.
• Derives closed-form expressions for posterior covariance and mean under the coreset model, enabling exact computation in exponential family models.
• Applies Riemannian geometry to unify past coreset methods (e.g., Hilbert coreset) as operating within a single tangent space, while the proposed method evolves along the manifold.
• Introduces a learning rate schedule $\gamma_t = t^{-1}$ for the greedy optimization, ensuring convergence in the iterative selection process.

Experimental results

Research questions

• RQ1 Can coreset construction be fully automated without requiring a user-specified coarse posterior approximation?
• RQ2 Does operating along the Riemannian manifold of the coreset posterior lead to better posterior approximation than fixed-tangent methods?
• RQ3 Can sparse variational inference in the exponential family enable unbiased and tractable KL divergence gradient estimation for coreset optimization?
• RQ4 How does the proposed method compare quantitatively in terms of KL divergence to the exact posterior across diverse models and datasets?
• RQ5 What is the computational cost of the proposed method relative to existing coreset construction techniques?

Key findings

• The proposed Riemannian coreset construction algorithm achieves orders-of-magnitude lower KL divergence to the exact posterior compared to past methods, especially beyond coreset sizes of 150.
• Past methods like GIGA are fundamentally limited by their fixed tangent plane approximation, which restricts posterior approximation quality as coreset size increases.
• Even with a realistic choice of coarse posterior, past methods such as GIGA perform worse than uniform subsampling, highlighting their sensitivity to input quality.
• The proposed method outperforms GIGA (Optimal) in KL divergence reduction without requiring the unrealistic assumption that $\hat{\pi} \approx \pi_1$.
• The method is fully automated, requiring only the probabilistic model and dataset as inputs, with no problem-specific tuning.
• Despite being slower than Hilbert coreset construction due to iterative gradient updates, SparseVI is the only practical method that significantly reduces KL divergence for reasonable coreset sizes when the optimal posterior is unavailable.