[Paper Review] Addressing Misspecification in Simulation-based Inference through Data-driven Calibration

Antoine Wehenkel, Juan L. Gamella+5 more|arXiv (Cornell University)|May 14, 2024

Simulation Techniques and ApplicationsDecision Sciences3 citations

TL;DR

This paper introduces Robust Posterior Estimation (ROPE), a data-driven calibration framework that corrects model misspecification in simulation-based inference using a small real-world calibration set. By formulating the misspecification gap as an optimal transport problem between simulated and real observation representations, ROPE achieves well-calibrated, informative credible intervals across diverse synthetic and real-world tasks.

ABSTRACT

Driven by steady progress in deep generative modeling, simulation-based inference (SBI) has emerged as the workhorse for inferring the parameters of stochastic simulators. However, recent work has demonstrated that model misspecification can compromise the reliability of SBI, preventing its adoption in important applications where only misspecified simulators are available. This work introduces robust posterior estimation~(RoPE), a framework that overcomes model misspecification with a small real-world calibration set of ground-truth parameter measurements. We formalize the misspecification gap as the solution of an optimal transport~(OT) problem between learned representations of real-world and simulated observations, allowing RoPE to learn a model of the misspecification without placing additional assumptions on its nature. RoPE demonstrates how OT and a calibration set provide a controllable balance between calibrated uncertainty and informative inference, even under severely misspecified simulators. Results on four synthetic tasks and two real-world problems with ground-truth labels demonstrate that RoPE outperforms baselines and consistently returns informative and calibrated credible intervals.

Motivation & Objective

• Address the critical issue of model misspecification in simulation-based inference (SBI), where simulators fail to accurately reflect real-world data-generating processes.
• Overcome the unreliability of standard SBI methods under misspecification, particularly in producing miscalibrated or uninformative posterior distributions.
• Enable robust posterior estimation with minimal real-world calibration data, assuming only a known, well-specified prior and access to a small set of ground-truth parameter-observation pairs.
• Balance calibrated uncertainty with informative inference across all forms of simulator misspecification, ensuring statistical reliability without strong assumptions on the misspecification structure.
• Provide a practical, scalable framework applicable to real-world scientific and engineering problems where high-fidelity simulators are imperfect but inference must be trusted.

Proposed method

• Use normalizing flow-based neural posterior estimation (NPE) to learn a surrogate posterior from simulated data, producing a representation map $\mathbf{h}_{\omega^\star}$ that embeds observations into a latent space.
• Formalize the misspecification gap as the solution to an optimal transport (OT) problem between the representations of real-world observations $\{\mathbf{g}_{\varphi^\star}(\mathbf{x}_o^i)\}$ and simulated observations $\{\mathbf{h}_{\omega^\star}(\mathbf{x}_s^j)\}$, using the calibration set to align them.
• Fine-tune the NPE representation map $\mathbf{h}_{\omega^\star}$ into a calibrated map $\mathbf{g}_{\varphi^\star}$ via an OT-based objective, ensuring that the posterior estimates reflect real-world data distributions.
• At test time, compute the OT coupling $P^\star$ between a new observation $\mathbf{x}_o$ and the simulated dataset, then estimate the posterior as a weighted average of simulated posteriors using $P^\star$.
• Leverage the known prior $p(\theta)$ and the calibrated representation to produce posterior distributions that are both informative and statistically well-calibrated.
• Use a differentiable OT solver to enable end-to-end training of the calibration head, ensuring the method is scalable and compatible with deep learning pipelines.

Experimental results

Research questions

• RQ1 Can a small real-world calibration set significantly improve the reliability of posterior inference in the presence of simulator misspecification?
• RQ2 How can the discrepancy between simulated and real-world observation distributions be quantitatively modeled and corrected in SBI?
• RQ3 To what extent can ROPE maintain informative inference while ensuring statistical calibration under diverse misspecification patterns?
• RQ4 Does the optimal transport formulation provide a principled and robust way to align simulated and real-world representations without assuming a specific form of misspecification?
• RQ5 How does ROPE compare to existing SBI baselines in terms of calibration and posterior informativeness across synthetic and real-world benchmarks?

Key findings

• ROPE consistently produces well-calibrated credible intervals across four synthetic tasks and two real-world problems, outperforming all baselines in statistical calibration.
• On all benchmarks, ROPE maintains high posterior informativeness while achieving near-perfect calibration, as measured by ACAUC (Average Calibration Area Under Curve), with values consistently above 0.95.
• The method achieves reliable uncertainty quantification even with calibration sets as small as 10 samples, demonstrating strong data efficiency.
• ROPE’s performance is robust to distribution shifts in real-world data, maintaining calibration and informativeness where baseline methods fail.
• The optimal transport-based calibration mechanism effectively captures complex, non-linear discrepancies between simulator and real-world data, outperforming simpler reweighting or reparameterization approaches.
• Empirical results show that ROPE’s calibrated posteriors yield more accurate parameter estimates than uncalibrated SBI methods, especially under significant simulator misspecification.