[Paper Review] Statistical Uncertainty Analysis for Stochastic Simulation

Wei Xie, Nelson, Barry L.+1 more|arXiv (Cornell University)|Nov 9, 2020

Simulation Techniques and Applications31 references22 citations

TL;DR

This paper proposes a metamodel-assisted bootstrapping method that quantifies total statistical uncertainty in stochastic simulations by jointly accounting for input uncertainty (from data-driven distribution estimation) and simulation uncertainty (from limited replications). Using stochastic kriging to model the response surface and bootstrap resampling to capture input variability, the approach produces a confidence interval that remains robust even under high metamodel uncertainty, with a variance decomposition that identifies whether reducing uncertainty requires more data or more simulations.

ABSTRACT

When we use simulation to evaluate the performance of a stochastic system, the simulation often contains input distributions estimated from real-world data; therefore, there is both simulation and input uncertainty in the performance estimates. Ignoring either source of uncertainty underestimates the overall statistical error. Simulation uncertainty can be reduced by additional computation (e.g., more replications). Input uncertainty can be reduced by collecting more real-world data, when feasible. This paper proposes an approach to quantify overall statistical uncertainty when the simulation is driven by independent parametric input distributions; specifically, we produce a confidence interval that accounts for both simulation and input uncertainty by using a metamodel-assisted bootstrapping approach. The input uncertainty is measured via bootstrapping, an equation-based stochastic kriging metamodel propagates the input uncertainty to the output mean, and both simulation and metamodel uncertainty are derived using properties of the metamodel. A variance decomposition is proposed to estimate the relative contribution of input to overall uncertainty; this information indicates whether the overall uncertainty can be significantly reduced through additional simulation alone. Asymptotic analysis provides theoretical support for our approach, while an empirical study demonstrates that it has good finite-sample performance.

Motivation & Objective

• To address the problem of underestimating total statistical error in stochastic simulations when both input and simulation uncertainties are present.
• To develop a confidence interval estimator that accounts for both input uncertainty (from finite real-world data) and metamodel uncertainty (from response surface approximation).
• To provide a variance decomposition measure that quantifies the relative contribution of input uncertainty to total uncertainty, guiding decisions on whether to collect more data or run more simulations.
• To ensure robust performance in finite samples, especially under tight computational budgets and unstable bootstrap moments.

Proposed method

• Input uncertainty is quantified using nonparametric bootstrap resampling of real-world data to generate multiple parameter estimates.
• A stochastic kriging metamodel is fitted to simulate outputs across the bootstrap-resampled parameter settings, propagating input uncertainty to the output mean.
• The metamodel uncertainty is derived from the predictive variance of the stochastic kriging model, which accounts for both estimation and prediction error.
• A combined confidence interval (CI+) is constructed that includes both input and metamodel uncertainty, using the properties of the metamodel and bootstrap variance.
• A variance decomposition is applied to estimate the relative contribution of input uncertainty to total uncertainty, defined as the ratio of input variance to total variance.
• Asymptotic consistency of the confidence interval is proven under the assumption that the true response surface is a Gaussian process with known parameters.

Experimental results

Research questions

• RQ1 Can a confidence interval be constructed that accounts for both input and metamodel uncertainty in stochastic simulation outputs?
• RQ2 How does the proposed method perform in finite samples when metamodel uncertainty is significant and computational budgets are limited?
• RQ3 To what extent does input uncertainty dominate total uncertainty, and can this be quantified to guide resource allocation between data collection and simulation runs?
• RQ4 Does the method maintain appropriate coverage probability even when bootstrap moments are unstable due to small real-world data samples?

Key findings

• The proposed confidence interval (CI+) maintains coverage close to the nominal level across various scenarios, including cases with high metamodel uncertainty, whereas the baseline method (CI0) fails when metamodel uncertainty is non-negligible.
• The variance decomposition ratio $\widehat{\sigma}_{I}/\widehat{\sigma}_{T}$ effectively estimates the relative contribution of input uncertainty to total uncertainty, approaching 1.0 when metamodel uncertainty is low and decreasing with more real-world data.
• When $m = 5000$ real-world data points, $\widehat{\sigma}_{I}/\widehat{\sigma}_{T}$ reached 0.974, indicating that input uncertainty was the dominant source of error, and both CI0 and CI+ achieved near-nominal coverage.
• Even with unstable bootstrap moments (e.g., $m = 50$), the method showed conservative behavior: 4.4% of CI0 and 3.8% of CI+ had lower bounds above the true mean, suggesting overcoverage rather than undercoverage.
• The method does not require sequential experimentation to reduce metamodel uncertainty, unlike prior approaches, making it suitable for computationally expensive simulations.
• Empirical results demonstrate good finite-sample performance on a complex problem with mixed discrete and continuous input distributions and tight computational constraints.