[Paper Review] Post-Inference Prior Swapping

Willie Neiswanger, Eric P. Xing|arXiv (Cornell University)|Jun 2, 2016

Gaussian Processes and Bayesian Inference3 citations

TL;DR

This paper introduces prior swapping, a method that enables efficient and accurate posterior inference under any target prior by transforming a pre-computed false posterior obtained using a convenient prior. The approach leverages data-independent updates to avoid costly re-running of inference algorithms, significantly outperforming importance sampling in terms of accuracy and speed, especially under dissimilar or complex priors.

ABSTRACT

While Bayesian methods are praised for their ability to incorporate useful prior knowledge, in practice, convenient priors that allow for computationally cheap or tractable inference are commonly used. In this paper, we investigate the following question: for a given model, is it possible to compute an inference result with any convenient false prior, and afterwards, given any target prior of interest, quickly transform this result into the target posterior? A potential solution is to use importance sampling (IS). However, we demonstrate that IS will fail for many choices of the target prior, depending on its parametric form and similarity to the false prior. Instead, we propose prior swapping, a method that leverages the pre-inferred false posterior to efficiently generate accurate posterior samples under arbitrary target priors. Prior swapping lets us apply less-costly inference algorithms to certain models, and incorporate new or updated prior information "post-inference". We give theoretical guarantees about our method, and demonstrate it empirically on a number of models and priors.

Motivation & Objective

• To address the challenge of efficiently computing posterior distributions under arbitrary target priors without re-running expensive inference algorithms.
• To overcome the limitations of importance sampling, which often fails due to high variance when the target prior differs significantly from the false prior.
• To enable post-inference incorporation of updated or new prior information in Bayesian models, particularly in models where standard inference is computationally prohibitive.
• To provide a general-purpose method that works across different types of inference results—exact densities, approximate posteriors, or MCMC samples—by transforming them to target posteriors.
• To demonstrate that prior swapping is both theoretically sound and empirically effective across diverse models, including generalized linear models and latent variable models with relational or sparsity-inducing priors.

Proposed method

• The method uses a pre-inferred false posterior $p_f(\theta|x^n)$ obtained under a convenient false prior $\pi_f(\theta)$, which allows for tractable or efficient inference.
• It applies data-independent iterative updates to transform samples or density estimates from $p_f(\theta|x^n)$ into accurate samples from the target posterior $p(\theta|x^n)$ under any desired prior $\pi(\theta)$.
• For MCMC samples, prior swapping performs a sequence of Metropolis-Hastings steps that adjust the proposal distribution based on the ratio of target to false prior densities.
• For parametric or approximate posterior densities, the method reweights or reparameterizes the false posterior using the ratio $\pi(\theta)/\pi_f(\theta)$ to align with the target prior.
• The approach is designed to be computationally efficient, as it avoids data-dependent updates and leverages the pre-computed inference result.
• Theoretical guarantees are provided for the convergence and accuracy of the method under various forms of false posterior representations.

Experimental results

Research questions

• RQ1 Can we transform a pre-computed false posterior obtained under a convenient prior into an accurate posterior under any arbitrary target prior without re-running inference?
• RQ2 How does prior swapping compare to importance sampling in terms of variance and accuracy when the target prior is very different from the false prior?
• RQ3 Can prior swapping efficiently handle complex, non-conjugate, or structured priors such as sparsity-inducing or relational priors over latent factors?
• RQ4 To what extent can prior swapping reduce computational cost in models where standard inference is expensive, such as in topic models or generalized linear models?
• RQ5 Does prior swapping maintain accuracy when the target prior is heavy-tailed or has a different parametric form than the false prior?

Key findings

• Prior swapping consistently achieves lower posterior estimation error than importance sampling, especially when the target prior is dissimilar to the false prior, where IS fails due to high variance.
• In Bayesian generalized linear models, prior swapping enables accurate inference under heavy-tailed and sparsity-inducing priors with orders of magnitude faster convergence than direct MCMC sampling.
• For latent factor models such as Gaussian mixture models and LDA, prior swapping successfully separates redundant topic clusters using a diversity-promoting prior, improving interpretability.
• Wall time measurements show that prior swapping is significantly faster than direct inference—by several orders of magnitude—especially when multiple target priors are considered.
• The method maintains high accuracy even when the target prior is highly non-Gaussian or has a different parametric form than the false prior, demonstrating robustness.
• Empirical results on synthetic and real-world data (e.g., Simple English Wikipedia corpus) confirm that prior swapping enables efficient posterior computation under complex priors without re-running costly inference pipelines.