[Paper Review] Bayesian model averaging via mixture model estimation

Merlin Keller, Kaniav Kamary|arXiv (Cornell University)|Nov 27, 2017

Statistical Methods and Bayesian Inference11 references3 citations

TL;DR

This paper proposes a Bayesian model averaging (BMA) method based on mixture model estimation, enabling the use of improper priors for shared parameters by sampling from a single encompassing mixture model. The approach uses MCMC to generate BMA posterior samples, from which model posterior probabilities and model-specific posterior expectations are efficiently estimated via Monte Carlo and importance sampling, with guaranteed low-variance estimates due to a self-pruning property.

ABSTRACT

A new approach for Bayesian model averaging (BMA) and selection is proposed, based on the mixture model approach for hypothesis testing in Kaniav et al., 2014. Inheriting from the good properties of this approach, it extends BMA to cases where improper priors are chosen for parameters that are common to all candidate models. From an algorithmic point of view, our approach consists in sampling from the posterior distribution of the single-datum mixture of all candidate models, weighted by their prior probabilities. We show that this posterior distribution is equal to the 'Bayesian-model averaged' posterior distribution over all candidate models, weighted by their posterior probability. From this BMA posterior sample, a simple Monte-Carlo estimate of each model's posterior probability is derived, as well as importance sampling estimates for expectations under each model's posterior distribution.

Motivation & Objective

• To address the challenge of Bayesian model averaging when improper priors are used for shared parameters across models.
• To provide a generic, computationally efficient alternative to Bayes factor computation in model selection and averaging.
• To enable direct sampling from the Bayesian model averaged posterior using a single MCMC run on an encompassing mixture model.
• To derive reliable Monte Carlo estimates of model posterior probabilities and importance sampling estimates of model-specific posteriors.
• To ensure well-behaved estimation by leveraging a self-pruning property that improves efficiency for high-posterior-probability models.

Proposed method

• Formulate Bayesian model averaging as a special case of mixture modeling by considering a single-datum mixture of all candidate models, weighted by their prior probabilities.
• Use MCMC to sample from the joint posterior distribution of the mixture model, including the model indicator and all model-specific parameters.
• Derive Monte Carlo estimates of each model’s posterior probability from the proportion of draws assigned to that model.
• Reconstruct model-specific posterior distributions using importance sampling based on the BMA posterior samples and model indicators.
• Ensure estimation stability by exploiting the self-pruning property, where the effective sample size of importance weights is proportional to the estimated posterior probability of each model.
• Allow improper priors for shared parameters as long as the resulting mixture posterior is proper, overcoming a key limitation of traditional BMA.

Experimental results

Research questions

• RQ1 Can Bayesian model averaging be reformulated as a mixture model estimation problem to allow improper priors for shared parameters?
• RQ2 How can posterior model probabilities and model-specific posterior expectations be efficiently estimated from a single MCMC run on an encompassing mixture model?
• RQ3 What are the finite-sample properties of the Monte Carlo and importance sampling estimates derived from the BMA posterior sample?
• RQ4 Does the proposed method exhibit robustness and efficiency, particularly for high-posterior-probability models?
• RQ5 What are the practical limitations of the method when the number of candidate models or parameter dimensionality is large?

Key findings

• The posterior distribution of the mixture model is mathematically equivalent to the Bayesian model averaged posterior, enabling exact BMA inference via mixture sampling.
• Monte Carlo estimates of model posterior probabilities are well-behaved and exhibit low variance, with effective sample size (ESS) proportional to the model’s posterior probability.
• Importance sampling estimates of model-specific posterior expectations are reliable, with tight confidence bounds observed in simulation studies.
• The method successfully recovers the true model in simulations, with BMA posterior estimates nearly as accurate as those from the true model.
• The approach allows the use of improper priors for shared parameters, a significant advantage over traditional BMA methods.
• The self-pruning property ensures that high-probability models are well-estimated, as their importance sampling weights maintain high effective sample size.