[Paper Review] Model Misspecification in ABC: Consequences and Diagnostics

David T. Frazier, Christian P. Robert+1 more|arXiv (Cornell University)|Aug 7, 2017

Markov Chains and Monte Carlo Methods20 references16 citations

TL;DR

This paper investigates the consequences of model misspecification in Approximate Bayesian Computation (ABC), demonstrating that different ABC variants—accept/reject ABC and regression-adjusted ABC—can concentrate posterior mass on distinct pseudo-true parameters under misspecification. It proposes diagnostics based on discrepancies between ABC versions to detect model misspecification, showing that such differences are systematic and detectable despite Monte Carlo variability.

ABSTRACT

We analyze the behavior of approximate Bayesian computation (ABC) when the model generating the simulated data differs from the actual data generating process; i.e., when the data simulator in ABC is misspecified. We demonstrate both theoretically and in simple, but practically relevant, examples that when the model is misspecified different versions of ABC can yield substantially different results. Our theoretical results demonstrate that even though the model is misspecified, under regularity conditions, the accept/reject ABC approach concentrates posterior mass on an appropriately defined pseudo-true parameter value. However, under model misspecification the ABC posterior does not yield credible sets with valid frequentist coverage and has non-standard asymptotic behavior. In addition, we examine the theoretical behavior of the popular local regression adjustment to ABC under model misspecification and demonstrate that this approach concentrates posterior mass on a completely different pseudo-true value than accept/reject ABC. Using our theoretical results, we suggest two approaches to diagnose model misspecification in ABC. All theoretical results and diagnostics are illustrated in a simple running example.

Motivation & Objective

• To analyze the theoretical behavior of ABC under model misspecification, particularly when the data simulator does not match the true data-generating process.
• To investigate how different ABC implementations—accept/reject ABC and local regression-adjusted ABC—behave under misspecification, revealing divergent asymptotic concentrations.
• To develop practical diagnostics for detecting model misspecification by exploiting systematic differences between ABC variants.
• To demonstrate that credible sets from misspecified ABC do not have valid frequentist coverage and exhibit non-standard asymptotic behavior.
• To evaluate the robustness of post-processing techniques like regression adjustment under model misspecification and suggest alternatives.

Proposed method

• Theoretical analysis of accept/reject ABC under model misspecification, showing posterior concentration on a pseudo-true parameter under regularity and identifiability conditions.
• Derivation of asymptotic behavior for regression-adjusted ABC, proving it concentrates on a different pseudo-true value than accept/reject ABC under misspecification.
• Proposed diagnostic procedure based on comparing posterior distributions from different ABC variants using a test statistic derived from the difference in parameter estimates.
• Use of Monte Carlo simulations to validate theoretical findings, comparing sampling distributions of differences between ABC outputs under various misspecification levels.
• Application of the diagnostic to a normal location-scale model with known misspecification, using summary statistics and tolerance-based comparisons.
• Calibration of diagnostic thresholds using reference tables generated from ABC output to distinguish genuine model misspecification from Monte Carlo noise.

Experimental results

Research questions

• RQ1 How does model misspecification affect the asymptotic behavior and concentration of ABC posteriors?
• RQ2 Do different ABC implementations—accept/reject ABC and regression-adjusted ABC—converge to the same pseudo-true parameter under model misspecification?
• RQ3 Can discrepancies between ABC variants be systematically exploited to detect model misspecification?
• RQ4 What is the frequentist coverage of credible sets derived from misspecified ABC posteriors?
• RQ5 How does the choice of tolerance sequence affect ABC performance under model misspecification compared to well-specified settings?

Key findings

• Under model misspecification, accept/reject ABC concentrates posterior mass on a pseudo-true parameter value defined by the closest distribution in the model class to the true data-generating process.
• Regression-adjusted ABC concentrates on a different pseudo-true parameter than accept/reject ABC, indicating that post-processing can exacerbate misspecification effects.
• Credible sets from ABC posteriors under misspecification do not have valid frequentist coverage, even asymptotically.
• The asymptotic behavior of ABC under misspecification is non-standard, deviating from typical posterior concentration patterns.
• The diagnostic procedure detects model misspecification with high power: 95–100% detection rate under $σ^2 = 2$ and $3$ in the normal example, regardless of the parameter used to construct the test statistic.
• The discrepancy between ABC variants, quantified via $\sqrt{n}\|\hat{h}-\tilde{h}\|$, is systematically larger under misspecification than under correct specification, enabling reliable detection.