[Paper Review] Using deterministic approximations to accelerate SMC for posterior sampling

Sophie Donnet, Stéphane Robin|arXiv (Cornell University)|Jul 25, 2017

Bayesian Methods and Mixture Models2 references3 citations

TL;DR

This paper proposes the Shortened Bridge Sampler (SBS), a novel Sequential Monte Carlo (SMC) method that accelerates posterior sampling by initializing from a deterministic posterior approximation—such as variational Bayes—instead of the prior. By adaptively constructing a bridge sequence between the approximate and true posterior, SBS corrects for underdispersion in variational approximations, yielding more accurate posterior variances and improved statistical inference, as demonstrated on network models where SBS revealed greater uncertainty than VB.

ABSTRACT

Sequential Monte Carlo has become a standard tool for Bayesian Inference of complex models. This approach can be computationally demanding, especially when initialized from the prior distribution. On the other hand, deter-ministic approximations of the posterior distribution are often available with no theoretical guaranties. We propose a bridge sampling scheme starting from such a deterministic approximation of the posterior distribution and targeting the true one. The resulting Shortened Bridge Sampler (SBS) relies on a sequence of distributions that is determined in an adaptive way. We illustrate the robustness and the efficiency of the methodology on a large simulation study. When applied to network datasets, SBS inference leads to different statistical conclusions from the one supplied by the standard variational Bayes approximation.

Motivation & Objective

• To address the computational burden of standard SMC methods that initialize from the prior, especially in high-dimensional or multimodal posterior distributions.
• To correct the well-known underestimation of posterior variances by variational Bayes (VB) approximations, which can lead to misleading inference.
• To develop a robust, adaptive SMC framework that leverages deterministic approximations as starting points to accelerate convergence without sacrificing accuracy.
• To provide a practical and efficient alternative to MCMC and standard SMC for complex Bayesian models where full posterior exploration is computationally prohibitive.

Proposed method

• Proposes a bridge sampling scheme that transitions from a deterministic posterior approximation to the true posterior using a sequence of tempered distributions.
• Constructs an adaptive sequence of distributions of the form $ \widetilde{p}_{\boldsymbol{Y}}^{1-\rho_h}(p(\theta|\boldsymbol{Y}))^{\rho_h} $, where $ \rho_h $ is dynamically adjusted during the SMC process.
• Uses sequential importance sampling with resampling to propagate particles from the approximate posterior toward the true posterior, ensuring the final sample is drawn from the exact posterior.
• Employs a path-finding strategy based on the quality of the initial approximation, with $ \rho_h $ values chosen to balance exploration and efficiency.
• Adapts the SMC algorithm to automatically determine the sequence of $ \rho_h $ values, reducing user tuning and improving robustness.
• Treats the deterministic approximation as a proposal mechanism rather than a prior, avoiding empirical Bayes pitfalls and enabling correction of structural misspecifications like ignored parameter dependencies.

Experimental results

Research questions

• RQ1 Can a deterministic posterior approximation be used to accelerate SMC sampling without compromising the accuracy of the final posterior sample?
• RQ2 How does the SBS method correct for the underestimation of posterior variances commonly observed in variational Bayes approximations?
• RQ3 To what extent does the SBS method recover parameter dependencies that are neglected in the initial deterministic approximation?
• RQ4 How does the performance of SBS compare to standard SMC and variational Bayes in terms of computational efficiency and statistical accuracy on complex models?
• RQ5 Can the adaptive sequence of tempering parameters $ \rho_h $ be reliably determined on the fly to ensure convergence and robustness?

Key findings

• On the Tree network dataset, SBS posterior standard deviations were nearly twice as large as those from variational Bayes, indicating a significant correction for underdispersion.
• For the regression coefficient $ \beta_1 $ (genetic distance), the posterior standard deviation increased from 1.5e-05 (VB) to 3.31e-05 (SBS), reducing the significance ratio from 3.09 to 1.25.
• In the Tree example, the posterior standard deviation for $ \beta_2 $ (geographic distance) increased from 0.208 (VB) to 0.47 (SBS), reducing its significance ratio from 1.11 to 0.755.
• For the SBM-reg model with one class, posterior probabilities were nearly identical between VB and SBS in most cases (e.g., Marriage, Business), but diverged dramatically in uncertain cases like Karate and Tree.
• In the Tree model, SBS estimated a posterior probability of $ 1.06 \times 10^{-161} $, while VB gave $ 4.83 \times 10^{-153} $, showing SBS is more conservative in detecting residual structure.
• The SBS algorithm successfully recovered parameter dependencies neglected in the initial VB approximation, as evidenced by the increased posterior variances and altered significance levels.