[Paper Review] Multivariate-from-Univariate MCMC Sampler: R Package MfUSampler

Alireza S. Mahani, Mansour T. A. Sharabiani|arXiv (Cornell University)|Dec 25, 2014

Markov Chains and Monte Carlo Methods9 references3 citations

TL;DR

The paper introduces the R package MfUSampler, which enables efficient multivariate Bayesian inference by combining univariate MCMC samplers—such as the slice sampler and adaptive rejection sampler—within an extended Gibbs sampling framework. By leveraging conditional distributions derived from the joint density and applying univariate Markov transitions, the method accelerates prototyping and allows incremental performance optimization through conjugacy, conditional independence, and porting to compiled languages.

ABSTRACT

The R package MfUSampler provides Monte Carlo Markov Chain machinery for generating samples from multivariate probability distributions using univariate sampling algorithms such as Slice Sampler and Adaptive Rejection Sampler. The sampler function performs a full cycle of univariate sampling steps, one coordinate at a time. In each step, the latest sample values obtained for other coordinates are used to form the conditional distributions. The concept is an extension of Gibbs sampling where each step involves, not an independent sample from the conditional distribution, but a Markov transition for which the conditional distribution is invariant. The software relies on proportionality of conditional distributions to the joint distribution to implement a thin wrapper for producing conditionals. Examples illustrate basic usage as well as methods for improving performance. By encapsulating the multivariate-from-univariate logic, MfUSampler provides a reliable library for rapid prototyping of custom Bayesian models while allowing for incremental performance optimizations such as utilization of conjugacy, conditional independence, and porting function evaluations to compiled languages.

Motivation & Objective

• Address the need for flexible, high-performance MCMC sampling in Bayesian modeling where DSLs like Stan or JAGS lack customization or efficiency.
• Overcome limitations of standard Gibbs sampling by replacing independent univariate draws with Markov transition steps that preserve conditional invariance.
• Provide a modular software library that decouples model specification from sampling logic, reducing implementation errors during prototyping.
• Enable incremental performance improvements through exploitation of conjugacy, conditional independence, and integration with high-performance languages like C/C++ or CUDA.
• Support researchers in building custom Bayesian models with full control over model structure while benefiting from robust, derivative-free univariate samplers.

Proposed method

• Implement an extended Gibbs sampling framework where each coordinate is updated via a univariate MCMC transition (e.g., slice or adaptive rejection sampling) instead of independent sampling.
• Use proportionality between the joint log-density and conditional distributions to compute conditionals without explicit analytical derivation.
• Encapsulate conditional density evaluation using wrapper functions (MfU.fEval, MfU.fgEval.f, MfU.fgEval.g) that compute the conditional log-density and gradients on-the-fly.
• Integrate two univariate samplers: the slice sampler with stepout and shrinkage (Neal, 2003), and adaptive rejection sampling (Gilks & Wild, 1992), with minimal modifications to existing R code.
• Provide a main sampling routine, MfU.Sample, that cycles through coordinates sequentially, updating each using the latest values of the others.
• Allow users to control sampler tuning parameters via MfU.Control, enabling customization of convergence behavior and performance.

Experimental results

Research questions

• RQ1 Can a modular software library be designed to enable efficient multivariate MCMC sampling using only univariate samplers, while preserving correctness and reducing implementation errors?
• RQ2 How does the performance of extended Gibbs sampling with univariate Markov transitions compare to standard independent sampling in Gibbs samplers for complex, correlated posterior distributions?
• RQ3 To what extent can performance be improved through incremental optimization strategies such as exploiting conjugacy, conditional independence, and porting log-density functions to compiled languages?
• RQ4 Can MfUSampler serve as a reliable foundation for prototyping complex Bayesian models before migrating to high-performance or parallelized code?
• RQ5 What is the impact of separating sampler logic from model specification on the maintainability and extensibility of Bayesian inference pipelines?

Key findings

• The MfUSampler package successfully enables multivariate MCMC sampling using only univariate samplers, with a runtime of 1.589 seconds for 1000 iterations in a high-dimensional Bayesian hierarchical model.
• The method allows researchers to rapidly prototype complex Bayesian models without reimplementing Gibbs sampling logic, reducing the risk of implementation errors.
• Performance gains are substantial when exploiting model structure: for hierarchical Bayesian regression, speedups scale approximately linearly with the number of groups due to conditional decomposition.
• The majority of computation time in large-scale models is dominated by log-density evaluations rather than sampling steps, making function porting to C/C++ or GPU code (e.g., CUDA) a high-impact optimization.
• The package serves as a blueprint for rewriting the core logic in high-performance languages, with minimal risk due to its clean abstraction of sampling control from model evaluation.
• The estimated posterior means from MfUSampler closely match true values (e.g., sigmamax ≈ 0.753 vs. true 0.750), confirming the method’s accuracy and convergence reliability.