[Paper Review] Does waste-recycling really improve Metropolis-Hastings Monte Carlo algorithm?

Jean‐François Delmas, Benjamin Jourdain|arXiv (Cornell University)|Nov 30, 2006

Markov Chains and Monte Carlo Methods12 references3 citations

TL;DR

This paper rigorously analyzes the waste-recycling (WR) Monte Carlo method, framing it as a control variate technique to reduce asymptotic variance in Metropolis-Hastings algorithms. It proves that while WR does not universally improve performance—contradicting common intuition—it is asymptotically superior in the Boltzmann algorithm setting, with optimal parameter estimation via empirical proposals.

ABSTRACT

The Metropolis Hastings algorithm and its multi-proposal extensions are aimed at the computation of the expectation $$ of a function $f$ under a probability measure $π$ difficult to simulate. They consist in constructing by an appropriate acceptation/rejection procedure a Markov chain $(X_k,k\geq 0)$ with transition matrix $P$ such that $π$ is reversible with respect to $P$ and in estimating $$ by the empirical mean $I_n(f)=\inv{n}\sum_{k=1}^n f(X_k)$. The waste-recycling Monte Carlo (WR) algorithm introduced by physicists is a modification of the Metropolis-Hastings algorithm, which makes use of all the proposals in the empirical mean, whereas the standard Metropolis-Hastings algorithm only uses the accepted proposals. In this paper, we extend the WR algorithm into a general control variate technique and exhibit the optimal choice of the control variate in terms of asymptotic variance. We also give an example which shows that in contradiction to the intuition of physicists, the WR algorithm can have an asymptotic variance larger than the one of the Metropolis-Hastings algorithm. However, in the particular case of the Metropolis-Hastings algorithm called Boltzmann algorithm, we prove that the WR algorithm is asymptotically better than the Metropolis-Hastings algorithm.

Motivation & Objective

• To rigorously evaluate whether waste-recycling (WR) Monte Carlo truly reduces asymptotic variance in multi-proposal Metropolis-Hastings algorithms.
• To formalize WR as a general control variate technique and identify the optimal control variate function that minimizes asymptotic variance.
• To resolve the long-standing assumption in physics literature that WR always improves efficiency by providing a counterexample where variance increases.
• To establish conditions under which WR is asymptotically better than standard Metropolis-Hastings, particularly in the Boltzmann algorithm framework.
• To propose a parametric estimator for the optimal WR parameter using empirical proposals in the multi-proposal setting.

Proposed method

• Reformulate the WR estimator as $ I_n(f) + J_n(eta) $, where $ J_n(eta) $ uses all proposals (accepted and rejected), treating it as a control variate problem.
• Derive the asymptotic variance of the WR estimator using the central limit theorem for Markov chains, showing dependence on the choice of control variate $ \psi $.
• Identify the optimal control variate $ F $ as the solution to the Poisson equation $ F - P F = f - \langle \pi, f \rangle $, which minimizes asymptotic variance.
• Propose a linear parametric generalization $ J_n(b f) $ of WR in the multi-proposal case and derive an estimator for the optimal $ b_* $ using the Markov chain path.
• Use the ergodic theorem and martingale techniques to prove almost sure convergence and asymptotic normality of the WR estimator under mild conditions.
• Construct a counterexample in the single-proposal case to show that WR can have larger asymptotic variance than standard Metropolis-Hastings.

Experimental results

Research questions

• RQ1 Does waste-recycling always reduce the asymptotic variance of the Metropolis-Hastings estimator, as commonly believed in physics literature?
• RQ2 Can the WR algorithm be formally framed as a control variate technique, and what is the optimal control variate function in terms of minimizing asymptotic variance?
• RQ3 In which specific cases—particularly the Boltzmann algorithm—does WR outperform standard Metropolis-Hastings in terms of asymptotic efficiency?
• RQ4 Is it possible to estimate the optimal parameter in a parametric family of WR estimators using only the Markov chain path?
• RQ5 What conditions ensure the consistency and asymptotic normality of the WR estimator when using a general control variate?

Key findings

• The WR algorithm does not universally improve performance: a counterexample in the single-proposal case shows it can yield larger asymptotic variance than standard Metropolis-Hastings.
• The optimal control variate in the WR framework is the solution $ F $ to the Poisson equation $ F - P F = f - \langle \pi, f \rangle $, which minimizes asymptotic variance.
• In the Boltzmann algorithm setting, the WR algorithm is asymptotically better than standard Metropolis-Hastings, with a provably smaller asymptotic variance.
• For the multi-proposal WR algorithm, the optimal parameter $ b_* $ in the parametric family $ J_n(b f) $ can be consistently estimated from the Markov chain path.
• The asymptotic variance of the WR estimator is given by $ \tilde{\sigma}(f,\beta)^2 = \sigma(f,\beta)^2 + \int \pi(dx) \left[ \text{Var}_{{\mathcal{Q}}(x,\cdot)}(\kappa\beta_x - \kappa F_x) - \text{Var}_{{\mathcal{Q}}(x,\cdot)}(\kappa F_x) \right] $, showing dependence on the choice of $ \beta $.
• The estimator $ I_n(f) + \mathcal{J}_n^\prime(\psi) $, where $ \mathcal{J}_n^\prime $ uses a modified selection kernel $ \kappa' $, is consistent under Harris recurrence and integrability conditions, even when $ \kappa' \neq \kappa $.