[Paper Review] Validation of Approximate Likelihood and Emulator Models for Computationally Intensive Simulations

Niccolò Dalmasso, Ann B. Lee+4 more|arXiv (Cornell University)|May 27, 2019

Gaussian Processes and Bayesian Inference68 references4 citations

TL;DR

This paper proposes a statistical framework for validating approximate likelihood and emulator models in computationally intensive simulations, using a regression-based two-sample test to detect distributional discrepancies and identify regions of poor fit. The method leverages permutation testing and links test power to regression model accuracy, enabling both global goodness-of-fit assessment and local diagnostics in parameter and feature space.

ABSTRACT

Complex phenomena in engineering and the sciences are often modeled with computationally intensive feed-forward simulations for which a tractable analytic likelihood does not exist. In these cases, it is sometimes necessary to estimate an approximate likelihood or fit a fast emulator model for efficient statistical inference; such surrogate models include Gaussian synthetic likelihoods and more recently neural density estimators such as autoregressive models and normalizing flows. To date, however, there is no consistent way of quantifying the quality of such a fit. Here we propose a statistical framework that can distinguish any arbitrary misspecified model from the target likelihood, and that in addition can identify with statistical confidence the regions of parameter as well as feature space where the fit is inadequate. Our validation method applies to settings where simulations are extremely costly and generated in batches or "ensembles" at fixed locations in parameter space. At the heart of our approach is a two-sample test that quantifies the quality of the fit at fixed parameter values, and a global test that assesses goodness-of-fit across simulation parameters. While our general framework can incorporate any test statistic or distance metric, we specifically argue for a new two-sample test that can leverage any regression method to attain high power and provide diagnostics in complex data settings.

Motivation & Objective

• To address the lack of consistent methods for validating approximate likelihood and emulator models in computationally intensive simulations.
• To distinguish misspecified models from the true likelihood with statistical confidence.
• To identify specific regions in parameter and feature space where emulator fits are inadequate.
• To provide a general framework applicable to various surrogate models, including Gaussian synthetic likelihoods and neural density estimators.
• To offer a goodness-of-fit test that answers whether a model is good enough, not just relative performance.

Proposed method

• Proposes a two-sample test using a regression-based approach to compare the true simulator distribution and the emulator distribution.
• Reformulates the two-sample test as a binary classification problem where Y indicates sample source (simulator vs. emulator).
• Uses the test statistic $ \widehat{\mathcal{T}} = \frac{1}{n}\sum_{i=1}^{n}(\widehat{m}(\mathbf{X}_i) - \widehat{\pi}_1)^2 $, where $ \widehat{m} $ estimates the probability of belonging to the emulator distribution.
• Employs a permutation procedure to compute p-values, ensuring valid inference without parametric assumptions.
• Links test power to the Mean Integrated Squared Error (MISE) of the regression estimator, ensuring high power when the regression model fits well.
• Enables local diagnostics by examining $ |\widehat{m}(\mathbf{x}) - \widehat{\pi}_1| $, which indicates local fit quality in feature space.

Experimental results

Research questions

• RQ1 Can we consistently validate approximate likelihood or emulator models when the true likelihood is intractable and simulations are computationally expensive?
• RQ2 How can we detect and localize discrepancies between the emulator and simulator distributions in parameter and feature space?
• RQ3 What statistical test can provide a global assessment of fit quality while maintaining high power in complex data settings?
• RQ4 How does the choice of regression method affect the power and reliability of the validation test?
• RQ5 Can we distinguish between a model that is good enough and one that requires further improvement, beyond relative loss metrics?

Key findings

• The proposed regression-based two-sample test effectively detects distributional differences between simulator and emulator outputs, even in high-dimensional or complex feature spaces.
• The test's power is directly linked to the MISE of the regression model used, meaning better regression performance leads to higher statistical power.
• Local discrepancies in the emulator fit are identified through the magnitude of $ |\widehat{m}(\mathbf{x}) - \widehat{\pi}_1| $, which highlights regions of poor fit in feature space.
• Permutation-based p-values ensure valid inference without requiring distributional assumptions, enhancing robustness.
• The framework successfully validates models in both synthetic and real-world cosmological data examples, demonstrating practical utility.
• The method outperforms standard loss functions (e.g., KL divergence) by providing an absolute measure of fit quality rather than relative performance.