[Paper Review] Population Empirical Bayes

Alp Kucukelbir, David M. Blei|arXiv (Cornell University)|Nov 2, 2014

Gaussian Processes and Bayesian Inference1 references4 citations

TL;DR

This paper introduces population empirical Bayes (pop-eb), a hierarchical Bayesian framework that improves predictive accuracy by modeling the empirical population distribution as a prior through a latent dataset. It uses bootstrap resampling and stochastic variational inference (bump-vi) to optimize predictive density, outperforming classical Bayesian inference in regression, mixture models, and topic models.

ABSTRACT

Bayesian predictive inference analyzes a dataset to make predictions about new observations. When a model does not match the data, predictive accuracy suffers. We develop population empirical Bayes (POP-EB), a hierarchical framework that explicitly models the empirical population distribution as part of Bayesian analysis. We introduce a new concept, the latent dataset, as a hierarchical variable and set the empirical population as its prior. This leads to a new predictive density that mitigates model mismatch. We efficiently apply this method to complex models by proposing a stochastic variational inference algorithm, called bumping variational inference (BUMP-VI). We demonstrate improved predictive accuracy over classical Bayesian inference in three models: a linear regression model of health data, a Bayesian mixture model of natural images, and a latent Dirichlet allocation topic model of scientific documents.

Motivation & Objective

• To address model mismatch in Bayesian inference, which degrades predictive accuracy when models do not reflect the true data-generating process.
• To develop a framework that integrates the empirical population distribution into Bayesian analysis without assuming model correctness.
• To improve predictive performance in complex models such as linear regression, Gaussian mixture models, and topic models.
• To provide a scalable, efficient inference method that maintains Bayesian coherence while being robust to model misspecification.

Proposed method

• Proposes a hierarchical model where the empirical population distribution is treated as a prior via a latent dataset variable.
• Uses bootstrap resampling to generate B datasets from the original data, each used to compute a posterior and predictive density.
• Selects the best-performing bootstrap dataset (X.b⇤)) based on predictive accuracy on the original data.
• Introduces bumping variational inference (bump-vi), a stochastic variational inference algorithm that efficiently computes gradients over bootstrap samples by reweighting local variational parameters.
• Applies local-global separation in variational inference: optimize local variables per observation and update global variables via stochastic gradient ascent.
• Uses subsampling equivalence to bootstrap resampling, enabling efficient gradient computation across bootstrap datasets without full retraining.

Experimental results

Research questions

• RQ1 Can modeling the empirical population distribution improve Bayesian predictive accuracy in misspecified models?
• RQ2 How does pop-eb compare to classical Bayesian inference when the assumed model is incorrect?
• RQ3 Can a scalable variational inference method be designed for population empirical Bayes that maintains predictive performance?
• RQ4 Does incorporating the population distribution via bootstrap resampling reduce brittleness in Bayesian predictions?
• RQ5 How does pop-eb perform across diverse models such as regression, mixture models, and topic models?

Key findings

• Pop-eb improves predictive accuracy over classical Bayesian inference in a linear regression model on health data, as shown in Table 1.
• In a Bayesian Gaussian mixture model of natural images, pop-eb’s predictive density better matches the true data distribution than standard Bayesian inference, as illustrated in Figure 5.
• For a latent Dirichlet allocation model on scientific documents, pop-eb’s predictive density aligns more closely with the empirical population than standard Bayesian inference, as shown in Figure 7.
• Bumping variational inference (bump-vi) enables efficient computation of pop-eb by reweighting local variational parameters across bootstrap samples, avoiding full recomputation.
• Even with a sharp prior, classical Bayesian inference fails to correct for model mismatch; pop-eb maintains superior predictive performance.
• Empirical Bayes priors estimated from data do not mitigate model mismatch, but pop-eb does so by explicitly modeling the population distribution.