[Paper Review] Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors

Yuling Yao, Aki Vehtari+1 more|arXiv (Cornell University)|Jun 22, 2020

Gaussian Processes and Bayesian Inference95 references34 citations

TL;DR

The paper proposes Bayesian stacking to combine parallel non-mixing inferences (MCMC, variational, or mode-based) to better represent multimodal posteriors and improve predictive performance, especially under model misspecification.

ABSTRACT

When working with multimodal Bayesian posterior distributions, Markov chain Monte Carlo (MCMC) algorithms have difficulty moving between modes, and default variational or mode-based approximate inferences will understate posterior uncertainty. And, even if the most important modes can be found, it is difficult to evaluate their relative weights in the posterior. Here we propose an approach using parallel runs of MCMC, variational, or mode-based inference to hit as many modes or separated regions as possible and then combine these using Bayesian stacking, a scalable method for constructing a weighted average of distributions. The result from stacking efficiently samples from multimodal posterior distribution, minimizes cross validation prediction error, and represents the posterior uncertainty better than variational inference, but it is not necessarily equivalent, even asymptotically, to fully Bayesian inference. We present theoretical consistency with an example where the stacked inference approximates the true data generating process from the misspecified model and a non-mixing sampler, from which the predictive performance is better than full Bayesian inference, hence the multimodality can be considered a blessing rather than a curse under model misspecification. We demonstrate practical implementation in several model families: latent Dirichlet allocation, Gaussian process regression, hierarchical regression, horseshoe variable selection, and neural networks.

Motivation & Objective

• Motivate the difficulty of inference with multimodal or metastable posteriors in Bayesian computation.
• Propose stacking as a scalable, parallelizable method to combine non-mixing chains to improve predictive performance.
• Extend stacking to combine multiple chains fitting the same model and provide practical implementation details.
• Analyze the asymptotic behavior showing stacking can outperform fully Bayesian inference under misspecification.
• Demonstrate the approach on diverse models to illustrate practical effectiveness.

Proposed method

• Run multiple parallel inference runs from dispersed starting points to explore multiple modes without relying on cross-chain mixing.
• Cluster or treat each run as a separate density p_k(θ|y) to be combined.
• Estimate leave-one-out predictive densities p_k(y_i|y_-i) for each run using Pareto smoothed importance sampling (PSIS).
• Solve a simplex-constrained optimization to find weights w that maximize the leave-one-out log predictive density (loo lpd) of the weighted mixture.
• Incorporate a Dirichlet-type regularization on weights to stabilize estimation and partially pool weights across chains.
• Plug the optimal weights into a weighted Monte Carlo form to approximate the target multimodal posterior.
• Provide practical steps for implementation, including monitoring convergence via lpd and optional clustering of chains.

Experimental results

Research questions

• RQ1 Can stacking of non-mixing, parallel inferences yield better predictive performance than any single chain or naive averaging?
• RQ2 How should we weight multiple non-mixing chains to best represent a multimodal posterior for prediction?
• RQ3 Is it possible for stacked predictions to outperform the exact Bayesian posterior under model misspecification?
• RQ4 How can we efficiently estimate leave-one-out predictive densities when chains do not mix?
• RQ5 What are practical guidelines for implementing stacking across different model families and computational settings?

Key findings

• Stacking provides weighted combinations of non-mixing chains that can improve predictive performance over uniform or single-chain estimates.
• An efficient importance-sampling-based approach (PSIS) can approximate leave-one-out predictive densities from full-data fits in each chain.
• The stacking weights maximize cross-validated predictive accuracy, yielding a posterior representation that remains multimodal in aggregate but better calibrated for prediction.
• Under model misspecification, stacked-chain inference can outperform the exact posterior in predictive terms in certain theoretical scenarios.
• The method is demonstrated across several model families, illustrating practical applicability to latent Dirichlet allocation, Gaussian process regression, hierarchical regression, horseshoe variable selection, and neural networks.