[Paper Review] Analyzing Stochastic Computer Models: A Review with Opportunities

Evan Baker, Pierre Barbillon+11 more|arXiv (Cornell University)|Feb 4, 2020

Advanced Multi-Objective Optimization Algorithms4 citations

TL;DR

This paper reviews statistical methods for analyzing stochastic computer models—simulators with inherent randomness due to pseudo-random number generation—focusing on Gaussian process emulators and their extensions. It presents practical approaches for design, calibration, and uncertainty quantification in stochastic simulators, emphasizing open challenges and opportunities in modeling non-constant noise, multi-fidelity frameworks, and variance reduction.

ABSTRACT

In modern science, computer models are often used to understand complex phenomena, and a thriving statistical community has grown around analyzing them. This review aims to bring a spotlight to the growing prevalence of stochastic computer models -- providing a catalogue of statistical methods for practitioners, an introductory view for statisticians (whether familiar with deterministic computer models or not), and an emphasis on open questions of relevance to practitioners and statisticians. Gaussian process surrogate models take center stage in this review, and these, along with several extensions needed for stochastic settings, are explained. The basic issues of designing a stochastic computer experiment and calibrating a stochastic computer model are prominent in the discussion. Instructive examples, with data and code, are used to describe the implementation of, and results from, various methods.

Motivation & Objective

• To provide practitioners and statisticians with accessible, up-to-date statistical tools for analyzing stochastic computer models, especially those with non-constant noise and inherent randomness.
• To highlight the limitations of standard regression and deterministic emulation methods when applied to stochastic simulators, particularly in high-dimensional or complex settings.
• To identify and emphasize unresolved research questions in design, calibration, and uncertainty quantification for stochastic simulators, especially in agent-based and complex simulation contexts.
• To promote the use of advanced methods such as multi-fidelity modeling, variance reduction techniques, and hybrid models combining deterministic and stochastic simulators.
• To encourage statistical research in stochastic simulator analysis by cataloging current methods, identifying gaps, and providing illustrative examples with data and code.

Proposed method

• Uses Gaussian process (GP) surrogate models as the central framework for modeling the mean response and residual variance in stochastic simulators, extending standard GP methods to handle non-constant noise.
• Applies hierarchical modeling strategies to jointly model the mean function $ M(x) $ and the variance $ \sigma_v^2(x) $, enabling flexible uncertainty quantification.
• Employs design strategies that balance replication and input space exploration, recognizing the trade-off between replicates and coverage in stochastic experiments.
• Introduces multi-fidelity modeling approaches that combine low-fidelity and high-fidelity simulators to improve efficiency and accuracy under limited computational budgets.
• Leverages variance reduction techniques and information from less noisy outputs to improve estimation of more variable quantities, such as quantiles.
• Utilizes replication data to estimate and model the heteroscedastic noise structure $ \sigma_v^2(x) $, avoiding the assumption of constant variance.

Experimental results

Research questions

• RQ1 How can Gaussian process emulators be adapted to handle non-constant variance in stochastic computer models, and what are the implications for uncertainty quantification?
• RQ2 What are effective design strategies for stochastic computer experiments that balance replication and input space coverage, especially under computational constraints?
• RQ3 How can model discrepancy be properly accounted for in the calibration of stochastic simulators, and which methods are most effective under different conditions?
• RQ4 In what ways can multi-fidelity modeling and variance reduction techniques improve the efficiency and accuracy of stochastic simulator analysis?
• RQ5 What are the key challenges in applying deep learning or neural networks to stochastic simulators, and how might they complement or extend Gaussian process methods?

Key findings

• Gaussian process emulators are effective for modeling the mean and variance of stochastic simulators, but diagnosing model inadequacy in stochastic settings remains underdeveloped compared to deterministic cases.
• Replication is essential for estimating noise variance $ \sigma_v^2(x) $, but the trade-off between replication and input space exploration complicates experimental design, especially in high dimensions.
• Current methods for calibration of stochastic simulators lack a one-size-fits-all solution, and empirical comparisons across methods (e.g., ABC vs. hierarchical modeling) are needed to guide best practices.
• Multi-fidelity modeling and coupling of deterministic and stochastic simulators offer viable paths to improve efficiency when simulation budgets are limited.
• Variance reduction techniques, though well-established in other contexts, are underutilized in agent-based models and stochastic simulators, suggesting a promising research direction.
• The use of less noisy outputs to inform modeling of noisier outputs—e.g., using the mean to improve quantile estimation—can enhance accuracy and is supported by empirical results in the literature.