[Paper Review] Validating Bayesian Inference Algorithms with Simulation-Based Calibration

Sean Talts, Michael Betancourt+3 more|arXiv (Cornell University)|Apr 18, 2018

Gaussian Processes and Bayesian Inference26 references220 citations

TL;DR

The paper introduces Simulation-Based Calibration (SBC) to validate Bayesian inference algorithms by checking whether rank statistics of priors vs posteriors are uniformly distributed, aiding detection of mis-specifications and algorithmic failures.

ABSTRACT

Verifying the correctness of Bayesian computation is challenging. This is especially true for complex models that are common in practice, as these require sophisticated model implementations and algorithms. In this paper we introduce \emph{simulation-based calibration} (SBC), a general procedure for validating inferences from Bayesian algorithms capable of generating posterior samples. This procedure not only identifies inaccurate computation and inconsistencies in model implementations but also provides graphical summaries that can indicate the nature of the problems that arise. We argue that SBC is a critical part of a robust Bayesian workflow, as well as being a useful tool for those developing computational algorithms and statistical software.

Motivation & Objective

• Motivate the need for validating Bayesian computation beyond single-run results.
• Propose SBC as a general, data-agnostic validation tool for any sampler producing posterior samples.
• Explain how SBC leverages the Bayesian joint distribution to detect implementation or computational errors.
• Provide guidance on interpreting SBC outputs and integrating SBC into a robust Bayesian workflow.

Proposed method

• Define the Bayesian joint distribution and a self-consistency ground truth by sampling from the prior and data from the model.
• Use rank statistics of one-dimensional functionals of parameters relative to posterior samples to form histograms expected to be uniform under correct computation (Theorem 1).
• Develop Algorithm 1 to generate SBC histograms of rank statistics across replicated simulated datasets.
• Address autocorrelation in posterior samples via thinning and effective sample size adjustments (Algorithm 2).
• Explain how deviations in SBC histograms map to over/under-dispersion and bias in posterior computations (visual diagnostics).
• Discuss extensions for small deviations and alternative visualizations (ECDF, binning strategies).

Experimental results

Research questions

• RQ1 Can SBC detect when a Bayesian analysis is mis-specified or mis-implemented?
• RQ2 How do deviations in SBC histograms relate to specific calibration failures (e.g., mis-specified priors, biased MCMC, approximate inference methods)?
• RQ3 How can SBC be adapted to correlated posterior samples from MCMC and to small deviations in practice?
• RQ4 What is the role of SBC within a robust Bayesian workflow alongside posterior predictive checks?
• RQ5 How do different inference technologies (e.g., HMC, ADVI, INLA) perform under SBC across representative models?

Key findings

• SBC reveals mis-specification such as using a different prior than used to generate data, producing a characteristic non-uniform rank histogram.
• SBC detects biased or autocorrelated posterior samples in MCMC methods, evidenced by distinct histogram shapes (e.g., cap, cup) under various failures.
• ADVI can fail by under- or overestimating posterior dispersion, identifiable via SBC histograms.
• INLA shows slight bias in a spatial disease prevalence model, detectable with SBC.
• Thinning and effective sample size adjustments mitigate autocorrelation effects, restoring SBC interpretability in many cases.
• SBC provides actionable insight into algorithmic and model implementation issues, complementing PPCs in a robust Bayesian workflow.