[Paper Review] Dimension-free Mixing for High-dimensional Bayesian Variable Selection

Quan Zhou, Jun Yang+3 more|arXiv (Cornell University)|May 12, 2021

Markov Chains and Monte Carlo Methods50 references4 citations

TL;DR

This paper proposes LIT-MH, a novel informed Metropolis-Hastings MCMC sampler for high-dimensional Bayesian variable selection that uses locally informed, thresholded proposal distributions to achieve dimension-free mixing—meaning its mixing time is independent of the number of covariates $p$. Under mild high-dimensional assumptions, the method achieves faster mixing than uninformed samplers while justifying the computational cost of local posterior evaluation.

ABSTRACT

Yang et al. (2016) proved that the symmetric random walk Metropolis--Hastings algorithm for Bayesian variable selection is rapidly mixing under mild high-dimensional assumptions. We propose a novel MCMC sampler using an informed proposal scheme, which we prove achieves a much faster mixing time that is independent of the number of covariates, under the same assumptions. To the best of our knowledge, this is the first high-dimensional result which rigorously shows that the mixing rate of informed MCMC methods can be fast enough to offset the computational cost of local posterior evaluation. Motivated by the theoretical analysis of our sampler, we further propose a new approach called "two-stage drift condition" to studying convergence rates of Markov chains on general state spaces, which can be useful for obtaining tight complexity bounds in high-dimensional settings. The practical advantages of our algorithm are illustrated by both simulation studies and real data analysis.

Motivation & Objective

• To address the challenge of slow mixing in high-dimensional Bayesian variable selection using MCMC, especially when the number of covariates $p$ is large.
• To rigorously demonstrate that informed MCMC methods can achieve mixing rates fast enough to offset their higher computational cost per iteration.
• To develop a new theoretical framework—'two-stage drift condition'—for analyzing convergence rates of Markov chains in high-dimensional settings.
• To design a practical, efficient MCMC sampler that leverages local posterior information without incurring dimension-dependent mixing slowdowns.
• To provide a theoretically grounded, dimension-free alternative to existing informed samplers that often suffer from poor mixing or require strong assumptions.

Proposed method

• Proposes LIT-MH (Locally Informed and Thresholded Metropolis-Hastings), which uses a balancing function $f(b) = \sqrt{b}$ to weight proposal probabilities based on the ratio $\pi_n(\gamma')/\pi_n(\gamma)$.
• Introduces a thresholded proposal mechanism that stabilizes the proposal distribution by truncating extreme values of the posterior ratio, improving numerical stability and mixing.
• Employs a two-stage drift condition to prove dimension-free mixing, a novel theoretical tool for bounding convergence rates on general state spaces.
• Uses add-delete-swap moves within the neighborhood $\mathcal{N}_1(\gamma)$, with proposal weights derived from local posterior density ratios.
• Applies a 'stabilized' version $\tilde{\mathbf{K}}_{\text{lb}}$ of the proposal kernel to avoid numerical instability from unbounded posterior ratios.
• Theoretical analysis shows that under Yang et al. (2016) assumptions, the mixing time of LIT-MH is independent of $p$, unlike standard random-walk MH.

Experimental results

Research questions

• RQ1 Can informed MCMC samplers for high-dimensional variable selection achieve mixing times independent of the number of covariates $p$?
• RQ2 Is the computational cost of evaluating local posterior densities justified by a sufficiently fast mixing rate in high-dimensional settings?
• RQ3 Can a new theoretical framework—'two-stage drift condition'—be developed to establish tight convergence bounds in high-dimensional Markov chain analysis?
• RQ4 Why do certain informed proposal schemes (e.g., LB-MH-1) perform better than others in practice, despite similar design principles?
• RQ5 How does LIT-MH compare to existing informed samplers like tempered Gibbs or ASI in terms of mixing speed and robustness?

Key findings

• LIT-MH achieves a mixing time that is independent of the number of covariates $p$, making it the first such result for an informed MCMC method in a general high-dimensional setting.
• The theoretical analysis introduces a new 'two-stage drift condition' that enables tighter convergence rate bounds in high-dimensional state spaces.
• Simulation studies show LIT-MH significantly outperforms the uninformed random-walk Metropolis-Hastings algorithm, even in multimodal posterior scenarios.
• The stabilized proposal version $\tilde{\mathbf{K}}_{\text{lb}}$ is shown to be numerically more stable and always preferable to the original $\mathbf{K}_{\text{lb}}$ in practice.
• LB-MH-1, which separates addition and deletion move weighting, avoids getting trapped in low-probability models due to high acceptance probabilities when the signal-to-noise ratio is sufficient.
• LIT-MH maintains strong performance without requiring adaptation or parallel chains, and its design allows for easy integration with tempering or blocking strategies.