[Paper Review] Light and Widely Applicable MCMC: Approximate Bayesian Inference for Large Datasets

Florian Maire, Nial Friel+1 more|arXiv (Cornell University)|Mar 13, 2015

Markov Chains and Monte Carlo Methods39 references3 citations

TL;DR

This paper introduces Light and Widely Applicable MCMC (LWA-MCMC), a novel approximate Bayesian inference method that uses a fixed, data-driven subset of observations to simulate a noisy Metropolis-Hastings kernel. By leveraging summary statistics to guide sub-sampling, LWA-MCMC achieves significant computational savings while maintaining strong theoretical guarantees and outperforming existing methods on large datasets, especially in i.i.d. exponential family models.

ABSTRACT

Light and Widely Applicable (LWA-) MCMC is a novel approximation of the Metropolis-Hastings kernel targeting a posterior distribution defined on a large number of observations. Inspired by Approximate Bayesian Computation, we design a Markov chain whose transition makes use of an unknown but fixed, fraction of the available data, where the random choice of sub-sample is guided by the fidelity of this sub-sample to the observed data, as measured by summary (or sufficient) statistics. LWA-MCMC is a generic and flexible approach, as illustrated by the diverse set of examples which we explore. In each case LWA-MCMC yields excellent performance and in some cases a dramatic improvement compared to existing methodologies.

Motivation & Objective

• To address the computational infeasibility of standard MCMC methods on large datasets where full likelihood evaluation is prohibitively expensive.
• To develop a generic, flexible, and widely applicable MCMC framework that maintains the simplicity of the Metropolis-Hastings algorithm while enabling scalability to big data.
• To reduce computational cost by using only a fixed fraction of data per iteration, guided by sub-sampling fidelity to observed data via summary statistics.
• To ensure theoretical validity by bounding the Kullback-Leibler divergence between the true posterior and the approximate posterior induced by sub-sampling.
• To demonstrate superior performance over existing noisy MCMC methods that often require large sub-samples to achieve confidence in acceptance decisions.

Proposed method

• LWA-MCMC constructs a noisy Metropolis-Hastings kernel by approximating the full-data likelihood ratio using a fixed-size, randomly selected subset $ U $ of the data, where $ |U| = n $.
• The subset $ U $ is selected based on its fidelity to the full data, measured by the closeness of its summary statistics $ \sum_{k \in U} S(Y_k) $ to the full-data sufficient statistics $ \sum_{k=1}^N S(Y_k) $.
• The acceptance probability is computed using a sub-sampled likelihood ratio, forming a noisy transition kernel that approximates the true posterior $ \pi $.
• The method ensures that the stationary distribution of the Markov chain is close to the true posterior by minimizing the KL divergence $ \text{KL}(\pi \| \tilde{\pi}_U) $, which is bounded using concentration and moment inequalities.
• Theoretical analysis shows that minimizing $ \| \xi_U \| = \left\| \frac{1}{n} \sum_{k \in U} S(Y_k) - \frac{1}{N} \sum_{k=1}^N S(Y_k) \right\| $ leads to tighter bounds on the approximation error.
• The framework is applicable to exponential family models and extends naturally to non-i.i.d. settings through sufficient statistics.

Experimental results

Research questions

• RQ1 Can a fixed, data-driven subset of observations be used to construct a valid, scalable MCMC sampler for large Bayesian models?
• RQ2 How can sub-sampling be guided to minimize the approximation error between the true posterior and the sub-sampled posterior?
• RQ3 Does LWA-MCMC achieve better computational efficiency than existing noisy MCMC methods without sacrificing accuracy?
• RQ4 What theoretical guarantees can be provided for the stationary distribution of the resulting Markov chain under sub-sampling?
• RQ5 Can the method be generalized beyond i.i.d. data and exponential family models?

Key findings

• LWA-MCMC achieves a dramatic reduction in computational cost by using only a fixed, small fraction of the data per iteration, avoiding the need to process all $ N $ observations.
• The method significantly outperforms existing noisy MCMC approaches such as those in Korattikara et al. (2014) and Bardenet et al. (2014), which often require large sub-samples to achieve confidence in decisions.
• Theoretical analysis shows that the KL divergence between the true posterior $ \pi $ and the approximate posterior $ \tilde{\pi}_U $ is bounded, with the bound minimized when the sub-sample's summary statistics are close to the full data's.
• The bound on the KL divergence depends on the deviation $ \| \xi_U \| $, and minimizing this deviation through sub-sampling selection leads to tighter approximation and better mixing.
• Empirical results demonstrate excellent performance in parameter estimation and classification tasks, confirming the method’s practical utility on real-world large datasets.
• The approach is widely applicable beyond i.i.d. settings and can be extended to non-exponential family models with appropriate sufficient statistics.