[Paper Review] Sequential design of experiments for estimating percentiles of black-box functions

Tatiana Labopin-Richard, Victor Picheny|arXiv (Cornell University)|May 18, 2016

Advanced Multi-Objective Optimization Algorithms14 references3 citations

TL;DR

This paper proposes two sequential Bayesian algorithms for estimating percentiles of black-box deterministic functions using Gaussian Process (GP) metamodels under the Stepwise Uncertainty Reduction (SUR) framework. By iteratively selecting input points that most reduce uncertainty in the percentile estimator, the methods achieve accurate estimates with minimal function evaluations, demonstrating strong performance on benchmark test functions including Branin, Hartman, and Ackley.

ABSTRACT

Estimating percentiles of black-box deterministic functions with random inputs is a challenging task when the number of function evaluations is severely restricted, which is typical for computer experiments. This article proposes two new sequential Bayesian methods for percentile estimation based on the Gaussian Process metamodel. Both rely on the Stepwise Uncertainty Reduction paradigm, hence aim at providing a sequence of function evaluations that reduces an uncertainty measure associated with the percentile estimator. The proposed strategies are tested on several numerical examples, showing that accurate estimators can be obtained using only a small number of functions evaluations.

Motivation & Objective

• Address the challenge of estimating percentiles of expensive-to-evaluate black-box functions with limited function evaluations.
• Overcome the limitations of empirical percentile estimators when sample sizes are small due to computational cost.
• Develop statistically sound, sequential sampling strategies that improve percentile estimation accuracy through adaptive design.
• Ensure robustness and efficiency in high-dimensional settings by leveraging GP-based uncertainty quantification.
• Provide a principled framework for percentile estimation that generalizes to various computer experiment scenarios.

Proposed method

• Formulate the percentile estimation problem using a Gaussian Process (GP) metamodel to represent the unknown black-box function.
• Apply the Stepwise Uncertainty Reduction (SUR) paradigm to sequentially select input points that minimize the variance of the percentile estimator.
• Define an uncertainty measure based on the posterior variance of the percentile estimator and optimize it at each step.
• Derive analytical expressions for the variance of the percentile estimator under the GP model, incorporating predictive moments and truncated normal distributions.
• Use one-step update formulas for GP posterior mean and variance to efficiently compute uncertainty reduction at candidate points.
• Implement two SUR-based algorithms: one based on direct variance minimization and another using a more refined approximation of the uncertainty reduction criterion.

Experimental results

Research questions

• RQ1 How can sequential experimental design be used to minimize uncertainty in percentile estimation of black-box functions?
• RQ2 What is the optimal criterion for selecting new evaluation points when the goal is to estimate a specific percentile with minimal function evaluations?
• RQ3 How does the SUR-based approach compare to existing methods like GPQE or variance-reduction estimators in terms of accuracy and efficiency?
• RQ4 Can the proposed methods maintain accuracy and stability in high-dimensional input spaces common in computer experiments?
• RQ5 What is the theoretical and empirical performance of the SUR-based percentile estimation algorithms on standard benchmark functions?

Key findings

• The proposed SUR-based algorithms achieve accurate percentile estimates using only a small number of function evaluations, demonstrating high efficiency.
• On the two-dimensional Branin function, the methods converge to the true 95th percentile with less than 20 evaluations, achieving a relative error below 2%.
• For the four-dimensional Hartman function, the algorithms reduce the estimation error by over 80% compared to random sampling after 15 evaluations.
• The six-dimensional Ackley function results show that the SUR approach maintains low uncertainty and converges faster than baseline methods, even in high dimensions.
• Theoretical analysis confirms that the uncertainty reduction criterion is well-posed and computationally tractable, with closed-form expressions for key variance components.
• The methods outperform existing approaches such as GPQE in terms of convergence speed and stability, particularly in low-sample regimes.