[Paper Review] Computing Bayes: Bayesian Computation from 1763 to the 21st Century

Gael M. Martin, David T. Frazier+1 more|arXiv (Cornell University)|Apr 14, 2020

Gaussian Processes and Bayesian InferenceComputer Science268 references21 citations

TL;DR

This paper provides a comprehensive historical and technical review of Bayesian computation from Thomas Bayes' 1763 essay to modern methods in the 21st century. It frames all computational techniques—ranging from Laplace approximation and MCMC to variational inference and ABC—within a unified framework of posterior expectation estimation, emphasizing their evolution, interconnections, and current challenges in scalability and model misspecification.

ABSTRACT

The Bayesian statistical paradigm uses the language of probability to express uncertainty about the phenomena that generate observed data. Probability distributions thus characterize Bayesian analysis, with the rules of probability used to transform prior probability distributions for all unknowns - parameters, latent variables, models - into posterior distributions, subsequent to the observation of data. Conducting Bayesian analysis requires the evaluation of integrals in which these probability distributions appear. Bayesian computation is all about evaluating such integrals in the typical case where no analytical solution exists. This paper takes the reader on a chronological tour of Bayesian computation over the past two and a half centuries. Beginning with the one-dimensional integral first confronted by Bayes in 1763, through to recent problems in which the unknowns number in the millions, we place all computational problems into a common framework, and describe all computational methods using a common notation. The aim is to help new researchers in particular - and more generally those interested in adopting a Bayesian approach to empirical work - make sense of the plethora of computational techniques that are now on offer; understand when and why different methods are useful; and see the links that do exist, between them all.

Motivation & Objective

• To provide a chronological and conceptual overview of Bayesian computation from Bayes' original 1763 problem to modern high-dimensional inference.
• To unify diverse computational methods—deterministic, simulation-based, and approximation techniques—under a common mathematical framework of posterior expectation evaluation.
• To help new researchers understand the strengths, limitations, and appropriate use cases of modern Bayesian computational tools.
• To highlight emerging challenges in scalability and model misspecification, and to identify key research directions for future development.

Proposed method

• Formulates Bayesian inference as the evaluation of posterior expectations E(g(θ)|y) = ∫Θ g(θ)p(θ|y)dθ, where p(θ|y) ∝ p(y|θ)p(θ) is known only up to a normalizing constant.
• Classifies computational methods into three categories: deterministic integration, simulation-based methods (e.g., MCMC, importance sampling), and approximation techniques (e.g., Laplace, variational inference, ABC).
• Uses consistent notation across methods to emphasize structural similarities and facilitate comparison, especially between classical and modern approaches.
• Analyzes the role of the marginal likelihood p(y|M) = ∫Θ p(y|θ,M)p(θ|M)dθ as a key quantity for model comparison, computed via posterior expectation under the prior.
• Reviews recent advances in scalable computation, including methods for large datasets and high-dimensional models, and discusses computational implications of model misspecification.
• Examines extensions beyond standard likelihood-based inference, such as generalized posteriors using loss functions (e.g., bootstrap-based posteriors, generalized variational inference), and MCMC on moment-based manifolds.

Experimental results

Research questions

• RQ1 How have computational methods for Bayesian inference evolved from Bayes’ original problem in 1763 to modern high-dimensional problems?
• RQ2 What unifying principles underlie diverse Bayesian computational techniques, and how can they be systematically compared?
• RQ3 What are the computational challenges and solutions in scaling Bayesian methods to modern data sizes and dimensions?
• RQ4 How do model misspecification and non-likelihood-based inference affect the design and performance of Bayesian computational algorithms?
• RQ5 What are the implications of using loss functions instead of likelihoods in Bayesian updating, and how do these affect computational efficiency and validity?

Key findings

• The core challenge of Bayesian computation remains the evaluation of intractable integrals, particularly posterior expectations, which are central to inference, prediction, decision theory, and model choice.
• Despite the availability of analytical solutions in simple cases (e.g., conjugate priors), most real-world problems require numerical approximation due to intractable normalizing constants.
• Modern computational methods such as MCMC, variational inference, and ABC are all designed to approximate the same fundamental quantity: E(g(θ)|y), with varying trade-offs in accuracy, speed, and scalability.
• Scalability is now a dominant research focus, with 16 of 129 conference abstracts at Bayes Comp 2020 explicitly addressing scalability, and many more targeting large datasets or high-dimensional models.
• Model misspecification significantly impacts computational methods: ABC may fail to yield valid credible sets under misspecification, and VB can still concentrate on the KL-minimizing parameter even when the model is wrong.
• New computational paradigms—such as generalized variational inference and MCMC on moment-based manifolds—are emerging to handle non-likelihood-based inference, signaling a fundamental shift in computational design for robust Bayesian analysis.