[Paper Review] Parallel MCMC via Weierstrass Sampler.

Xiangyu Wang, David B. Dunson|arXiv (Cornell University)|Dec 17, 2013

Bayesian Methods and Mixture Models30 citations

TL;DR

This paper proposes the Weierstrass sampler, a communication-free parallel MCMC method that combines posterior samples from independent subset MCMC chains to approximate the full-data posterior efficiently. It bounds approximation error via tuning parameters and demonstrates superior performance over averaging and kernel smoothing in simulation studies.

ABSTRACT

With the rapidly growing scales of statistical problems, subset based communication-free parallel MCMC methods are a promising future for large scale Bayesian analysis. In this article, we propose a new Weierstrass sampler for parallel MCMC based on independent subsets. The new sampler approximates the full data posterior samples via combining the posterior draws from independent subset MCMC chains, and thus enjoys a higher computational efficiency. We show that the approximation error for the Weierstrass sampler is bounded by some tuning parameters and provide suggestions for choice of the values. Simulation study shows the Weierstrass sampler is very competitive compared to other methods for combining MCMC chains generated for subsets, including averaging and kernel smoothing.

Motivation & Objective

• To address the challenge of scaling Bayesian MCMC to large datasets by enabling efficient, communication-free parallel computation.
• To develop a method that combines posterior samples from independent subset chains without requiring inter-chain communication.
• To bound the approximation error of the combined posterior estimate using tunable parameters.
• To improve computational efficiency and accuracy in large-scale Bayesian inference compared to existing aggregation methods.
• To provide practical guidelines for selecting tuning parameters that control error and performance.

Proposed method

• The Weierstrass sampler constructs a weighted combination of posterior samples from independent MCMC chains fitted on disjoint data subsets.
• It leverages the Weierstrass transformation to smooth and combine posterior draws, approximating the full-data posterior density.
• The method uses a kernel-based weighting scheme that depends on tuning parameters controlling the trade-off between bias and variance.
• The approximation error is analytically bounded in terms of the tuning parameters and the geometry of the posterior distribution.
• The sampler operates without communication between chains, enabling high scalability on distributed systems.
• It supports efficient posterior inference by aggregating results from parallel, independent subset chains.

Experimental results

Research questions

• RQ1 How can posterior samples from independent subset MCMC chains be combined to approximate the full-data posterior with bounded error?
• RQ2 What tuning parameters control the approximation error in the combined posterior estimate?
• RQ3 How does the Weierstrass sampler compare in accuracy and efficiency to averaging and kernel smoothing methods for combining MCMC chains?
• RQ4 Can the method achieve high computational efficiency while maintaining statistical fidelity in large-scale Bayesian inference?
• RQ5 What practical guidelines exist for selecting tuning parameters to balance accuracy and computational cost?

Key findings

• The Weierstrass sampler achieves bounded approximation error that depends on the choice of tuning parameters, ensuring statistical reliability.
• The method demonstrates competitive performance compared to averaging and kernel smoothing in simulation studies, particularly in terms of accuracy and convergence.
• The sampler enables communication-free parallelization, significantly improving computational efficiency for large-scale datasets.
• Theoretical bounds on error are derived and shown to be effective in practice with appropriate parameter selection.
• Simulation results confirm that the Weierstrass sampler outperforms or matches existing methods in estimating posterior distributions from subset data.
• Practical suggestions for tuning parameter selection are provided, enhancing usability in real-world applications.