[Paper Review] Robust and reproducible model selection using bagged posteriors

Jonathan H. Huggins, Jeffrey W. Miller|arXiv (Cornell University)|Jul 24, 2020

Bayesian Methods and Mixture Models20 references4 citations

TL;DR

This paper proposes BayesBag, a robust model selection method that improves Bayesian model selection by averaging posterior model probabilities over bootstrapped datasets. Under model misspecification, BayesBag enhances reproducibility and accuracy in model selection while remaining slightly more conservative under correct specification.

ABSTRACT

Bayesian model selection is premised on the assumption that the data are
 generated from one of the postulated models, however, in many applications, all of
 these models are incorrect. When two or more models provide a nearly equally good
 t to the data, Bayesian model selection can be highly unstable, potentially leading
 to self-contradictory ndings. In this paper, we explore using bagging on the posterior
 distribution (\\BayesBag") when performing model selection { that is, averaging the
 posterior model probabilities over many bootstrapped datasets. We provide theoreti-
 cal results characterizing the asymptotic behavior of the standard posterior and the
 BayesBag posterior under misspeci cation, in the model selection setting. We empir-
 ically assess the BayesBag approach on synthetic and real-world data in (i) feature
 selection for linear regression and (ii) phylogenetic tree reconstruction. Our theory
 and experiments show that in the presence of misspeci cation, BayesBag provides
 (a) greater reproducibility and (b) greater accuracy in selecting the correct model,
 compared to the standard Bayesian posterior; on the other hand, under correct speci-
 cation, BayesBag is slightly more conservative than the standard posterior. Overall,
 our results demonstrate that BayesBag provides an easy-to-use and widely applicable
 approach that improves upon standard Bayesian model selection by making it more
 stable and reproducible.

Motivation & Objective

• To address the instability of standard Bayesian model selection when multiple models fit the data nearly equally well.
• To improve reproducibility in model selection under model misspecification.
• To develop a practical and widely applicable method that enhances stability without sacrificing accuracy.
• To theoretically characterize the asymptotic behavior of standard and bagged posteriors under misspecification.
• To empirically validate BayesBag on feature selection and phylogenetic tree reconstruction.

Proposed method

• Applies bagging to the posterior distribution by generating multiple bootstrapped datasets from the original data.
• Computes the posterior model probability for each bootstrapped dataset.
• Averages the posterior model probabilities across all bootstrapped datasets to form the BayesBag posterior.
• Theoretical analysis derives the asymptotic behavior of both standard and BayesBag posteriors under model misspecification.
• Empirically evaluates BayesBag on synthetic and real-world data in linear regression and phylogenetic tree selection.
• Uses the averaged posterior probabilities as the final model selection criterion.

Experimental results

Research questions

• RQ1 How does the standard Bayesian posterior behave under model misspecification in model selection?
• RQ2 Can bagging the posterior improve the reproducibility of model selection results?
• RQ3 Does BayesBag enhance accuracy in selecting the correct model when all candidate models are misspecified?
• RQ4 How does BayesBag compare to standard Bayesian model selection in terms of stability and performance?
• RQ5 What is the theoretical asymptotic behavior of the BayesBag posterior under misspecification?

Key findings

• BayesBag significantly improves reproducibility of model selection results when models are misspecified.
• BayesBag increases accuracy in selecting the correct model under model misspecification compared to standard Bayesian posterior.
• Under correct model specification, BayesBag is slightly more conservative than the standard posterior.
• Theoretical analysis confirms that BayesBag stabilizes posterior model probabilities under misspecification.
• Empirical results on feature selection and phylogenetic tree reconstruction demonstrate consistent improvements in stability and accuracy.
• BayesBag provides a practical, easy-to-implement enhancement to standard Bayesian model selection.