[Paper Review] Sequential design of experiments to estimate a probability of exceeding a threshold in a multi-fidelity stochastic simulator

Rémi Stroh, Séverine Demeyer+3 more|arXiv (Cornell University)|Jul 26, 2017

Advanced Multi-Objective Optimization Algorithms7 references3 citations

TL;DR

This paper proposes a Bayesian sequential design strategy, Maximum Speed of Uncertainty Reduction (MSUR), for estimating the probability that a multi-fidelity stochastic simulator exceeds a critical threshold at high fidelity. By adaptively selecting both physical inputs and fidelity levels, MSUR achieves uncertainty reduction comparable to the best single-level SUR strategies without prior knowledge of the optimal fidelity level, leveraging Gaussian process models and multi-fidelity kriging to balance accuracy and computational cost.

ABSTRACT

In this article, we consider a stochastic numerical simulator to assess the impact of some factors on a phenomenon. The simulator is seen as a black box with inputs and outputs. The quality of a simulation, hereafter referred to as fidelity, is assumed to be tunable by means of an additional input of the simulator (e.g., a mesh size parameter): high-fidelity simulations provide more accurate results, but are time-consuming. Using a limited computation-time budget, we want to estimate, for any value of the physical inputs, the probability that a certain scalar output of the simulator will exceed a given critical threshold at the highest fidelity level. The problem is addressed in a Bayesian framework, using a Gaussian process model of the multi-fidelity simulator. We consider a Bayesian estimator of the probability, together with an associated measure of uncertainty, and propose a new multi-fidelity sequential design strategy, called Maximum Speed of Uncertainty Reduction (MSUR), to select the value of physical inputs and the fidelity level of new simulations. The MSUR strategy is tested on an example.

Motivation & Objective

• To estimate the probability that a stochastic simulator's output exceeds a critical threshold at the highest fidelity level under a limited computational budget.
• To address the challenge of balancing simulation accuracy and computational cost in multi-fidelity stochastic simulators.
• To develop a sequential experimental design strategy that adaptively selects both physical inputs and fidelity levels to minimize uncertainty in the probability estimate.
• To provide a Bayesian framework that models the simulator output using Gaussian processes and quantifies uncertainty in the threshold exceedance probability.
• To demonstrate that the proposed MSUR strategy outperforms single-level SUR strategies without requiring prior knowledge of the optimal fidelity level.

Proposed method

• Models the simulator output as conditionally normal with mean and variance functions modeled via Gaussian processes, assuming known covariance and variance functions.
• Uses a Gaussian process prior over the mean function ξ(x,t), with an improper uniform prior on the mean, enabling conjugate posterior updates via kriging formulas.
• Derives the threshold exceedance probability p(x) as a cumulative normal CDF transformation of the GP posterior mean and standard deviation at high fidelity.
• Proposes a new sequential design criterion, Maximum Speed of Uncertainty Reduction (MSUR), which selects the next simulation point (x,t) to maximize the expected reduction in posterior variance of the probability of interest.
• Adapts the SUR (Sequential Uncertainty Reduction) framework to multi-fidelity settings by jointly optimizing over physical inputs and fidelity levels to accelerate convergence.
• Employs a multi-fidelity kriging model to borrow information across fidelity levels, improving estimation efficiency while respecting computational cost trade-offs.

Experimental results

Research questions

• RQ1 Can a sequential design strategy effectively reduce uncertainty in the probability of threshold exceedance for a stochastic multi-fidelity simulator?
• RQ2 How can the selection of both physical inputs and fidelity levels be optimized to accelerate uncertainty reduction in the probability estimate?
• RQ3 Does the proposed MSUR strategy achieve performance comparable to the best single-level SUR strategies without prior knowledge of the optimal fidelity level?
• RQ4 How does the multi-fidelity Bayesian framework with Gaussian process priors enable efficient estimation of rare-event probabilities in computationally expensive simulators?
• RQ5 What is the trade-off between computational cost and accuracy in threshold exceedance probability estimation across different fidelity levels?

Key findings

• The MSUR strategy achieves uncertainty reduction performance comparable to the best single-level SUR strategies across all tested fidelity levels, including the most accurate one.
• At a threshold exceedance probability error level of 0.05, MSUR performs as well as the SL-SUR strategy using the finest time step (Δ = 0.05 s), demonstrating robustness across fidelity levels.
• MSUR successfully identifies the most informative combinations of physical inputs and fidelity levels without requiring prior knowledge of which fidelity level offers the best accuracy-cost trade-off.
• The method significantly reduces the mean squared error in estimating both the global probability P and the local probability function p(x) across increasing computational budgets.
• The MSUR strategy outperforms single-level SUR strategies in terms of uncertainty reduction per unit cost, especially when high-fidelity simulations are expensive.
• Empirical results show that MSUR maintains low error in both L² norm for the probability function and for the global probability, even with limited simulation budget.