[Paper Review] Deviance Information Criteria for Model Selection in Approximate Bayesian Computation

Olivier François, Guillaume Laval|arXiv (Cornell University)|May 2, 2011

Bayesian Methods and Mixture Models40 references4 citations

TL;DR

This paper proposes a deviance information criterion (DIC) based on posterior predictive distributions for model selection in Approximate Bayesian Computation (ABC), addressing inconsistencies between model probability estimation and posterior predictive checks. By approximating the deviance using regression-adjusted ABC samples, the method provides a reliable, likelihood-free alternative to traditional ABC model choice that avoids reliance on approximate posterior model probabilities.

ABSTRACT

Approximate Bayesian computation (ABC) is a class of algorithmic methods in Bayesian inference using statistical summaries and computer simulations. ABC has become popular in evolutionary genetics and in other branches of biology. However model selection under ABC algorithms has been a subject of intense debate during the recent years. Here we propose novel approaches to model selection based on posterior predictive distributions and approximations of the deviance. We argue that this framework can settle some contradictions between the computation of model probabilities and posterior predictive checks using ABC posterior distributions. A simulation study and an analysis of a resequencing data set of human DNA show that the deviance criteria lead to sensible results in a number of model choice problems of interest to population geneticists.

Motivation & Objective

• Address the inconsistency between approximate posterior model probabilities and posterior predictive checks in ABC-based model selection.
• Overcome the limitations of standard ABC model choice that rely on rejection algorithm outputs without regression adjustments.
• Develop a model selection framework grounded in posterior predictive distributions that aligns with corrected parameter inferences.
• Provide a practical, computationally feasible alternative to model probability estimation in ABC, especially for complex population genetic models.
• Demonstrate the method's reliability through simulation studies and real human genomic data analysis.

Proposed method

• Define an approximate deviance based on the posterior predictive distribution of summary statistics under each model.
• Use regression-adjusted ABC samples to improve the approximation of the posterior distribution, ensuring consistency with parameter inference.
• Compute the expected deviance and deviance information criterion (DIC) using the posterior predictive distribution of the deviance.
• Apply the DIC to compare models without relying on marginal likelihood approximations or raw rejection counts.
• Utilize sequential Monte Carlo and iterative importance sampling to estimate posterior predictive quantities in complex models.
• Validate the method by comparing DIC values across models with varying prior distributions on key parameters, such as Neanderthal replacement rates.

Experimental results

Research questions

• RQ1 Can a DIC-based approach improve model selection reliability in ABC when standard posterior model probabilities are inconsistent with posterior predictive checks?
• RQ2 How does the use of regression-adjusted ABC samples affect the reliability of model selection compared to raw rejection-based estimates?
• RQ3 To what extent does the proposed deviance criterion perform in realistic population genetic models with complex demographic histories?
• RQ4 Does the DIC approach maintain consistency across different ABC algorithms, including rejection and sequential Monte Carlo methods?
• RQ5 Can the method detect model inconsistencies that are missed by traditional ABC model choice based on posterior probabilities?

Key findings

• The proposed DIC-based model selection method produces sensible results in model choice problems relevant to population genetics, particularly in scenarios with high replacement rates of Neanderthals by modern humans.
• For high values of the Neanderthal replacement rate δ, both the expected deviance D̄₁ and DIC₁ were close to each other, indicating model stability and consistency.
• Models with high δ and low levels of Neanderthal genetic introgression were favored by the deviance criteria, consistent with findings from Green et al. (2010).
• The simulation study demonstrated that the deviance-based approach effectively identifies model inconsistencies that may be missed by standard ABC model probability estimation.
• The method avoids the pitfalls of relying on unadjusted posterior model probabilities and instead uses regression-corrected posterior predictive distributions.
• Since DIC computation is based on posterior predictive distributions, the approach is applicable to any ABC algorithm, including rejection, regression-adjusted, and sequential Monte Carlo methods.