[Paper Review] Paired-move multiple-try stochastic search for Bayesian variable selection

Xu Chen, Shaan Qamar+1 more|arXiv (Cornell University)|Nov 29, 2016

Statistical Methods and Inference5 references3 citations

TL;DR

This paper proposes a novel paired-move multiple-try Metropolis (pMTM) MCMC algorithm for Bayesian variable selection in high-dimensional 'large p, small n' settings. By generalizing multiple-try Metropolis to discrete model spaces and integrating neighborhood-based stochastic search, the method enables efficient exploration of the model space with improved mixing and scalability, significantly reducing predictive mean squared error compared to existing methods while maintaining competitive model sizes.

ABSTRACT

Variable selection is a key issue when analyzing high-dimensional data. The explosion of data with large sample sizes and dimensionality brings new challenges to this problem in both inference accuracy and computational complexity. To alleviate these problems, we propose a new scalable Markov chain Monte Carlo (MCMC) sampling algorithm for "large $p$ small $n$" scenarios by generalizing multiple-try Metropolis to discrete model spaces and further incorporating neighborhood-based stochastic search. The proof of reversibility of the proposed MCMC algorithm is provided. Extensive simulation studies are performed to examine the efficiency of the new algorithm compared with existing methods. A real data example is provided to illustrate the prediction performances of the new algorithm.

Motivation & Objective

• To address the computational and inferential challenges of Bayesian variable selection in high-dimensional data with large p and small n.
• To improve mixing and convergence in MCMC exploration of the discrete model space by incorporating neighborhood-based stochastic search.
• To enhance scalability and efficiency through a paired-move multiple-try Metropolis framework that reduces reliance on local, sequential scanning.
• To provide a robust, reproducible method for variable selection with reliable uncertainty quantification via posterior model probabilities.
• To enable practical application on large-scale datasets through parallelization and computational budget control.

Proposed method

• Generalizes the multiple-try Metropolis algorithm to discrete model spaces by introducing paired moves that propose multiple candidate models simultaneously.
• Introduces a neighborhood-based stochastic search strategy that explores models in local neighborhoods, improving mixing in correlated predictor settings.
• Employs a reversible MCMC transition kernel with a carefully designed proposal distribution to ensure detailed balance and convergence to the target posterior.
• Uses a flexible computational budget parameter M, set to p/10 in experiments, to control the number of candidate models proposed per iteration.
• Implements parallelization via cluster-based computation using the parallel package in R, distributing data and evaluating marginal likelihoods across multiple cores.
• Applies adaptive tuning in ada-pMTM to update scores for predictors dynamically, reducing the number of marginal likelihood evaluations compared to standard pMTM.

Experimental results

Research questions

• RQ1 Can a multiple-try Metropolis algorithm be effectively generalized to discrete model spaces for Bayesian variable selection?
• RQ2 How does the paired-move strategy improve mixing and convergence in high-dimensional model space exploration compared to sequential scanning?
• RQ3 What is the impact of varying the computational budget M on the efficiency and accuracy of model selection?
• RQ4 Can parallelization significantly improve runtime without compromising sampling quality in large-scale variable selection?
• RQ5 How does the proposed method compare in predictive performance and model selection accuracy to existing approaches like Lasso, SCAD, and SSS?

Key findings

• The pMTM and ada-pMTM algorithms achieve lower predictive mean squared error (MSE) than competing methods, with ada-pMTM reporting 116.43 MSE on the real data example.
• pMTM and ada-pMTM produce smaller model sizes than Lasso and adaptive Lasso while maintaining superior predictive performance.
• The algorithm demonstrates strong scalability, with performance improving under parallelization when M is large, particularly with 8 cores.
• Communication overhead limits gains with 4 or 8 clusters when M = p/5, but computation time dominates as M increases, making parallelization beneficial.
• The method converges quickly to equilibrium with few samples, satisfying both convergence and sample size requirements for reproducibility.
• The hybrid approach combining pRNS and ada-pMTM is suggested as a promising direction for future work to balance sample diversity and convergence speed.