[Paper Review] Adaptive design of supercomputer experiments

Robert B. Gramacy, Herbert K. H. Lee|arXiv (Cornell University)|May 28, 2008

Machine Learning and Algorithms37 references16 citations

TL;DR

This paper proposes an adaptive sequential design framework for supercomputer experiments that uses a Bayesian treed Gaussian process to model nonstationary response surfaces while minimizing computational cost. By leveraging predictive uncertainty to guide experimental runs in an asynchronous, agent-based environment, the method improves spatial coverage and accuracy compared to fixed sparse designs.

ABSTRACT

Computer experiments are often performed to allow modeling of a response surface of a physical experi-ment that can be too costly or difficult to run except using a simulator. Running the experiment over a dense grid can be prohibitively expensive, yet running over a sparse design chosen in advance can result in obtaining insufficient information in parts of the space, particularly when the surface is nonstation-ary. We propose an approach which automatically explores the space while simultaneously fitting the response surface, using predictive uncertainty to guide subsequent experimental runs. The newly devel-oped Bayesian treed Gaussian process is used as the surrogate model, and a fully Bayesian approach allows explicit nonstationary measures of uncertainty. Our adaptive sequential design framework has been developed to cope with an asynchronous, random, agent-based supercomputing environment. We take a hybrid approach which melds optimal strategies from the statistics literature with flexible strate-gies from the active learning literature. The merits of this approach are borne out in several examples, including the motivating example of a computational fluid dynamics simulation of rocket booster. Key words: nonstationary spatial model, treed partitioning, sequential design, active learning 1

Motivation & Objective

• To address the high cost and inefficiency of dense grid sampling in supercomputer experiments.
• To overcome limitations of fixed sparse designs that may miss critical regions in nonstationary response surfaces.
• To develop a framework that adaptively explores the input space while simultaneously building accurate surrogate models.
• To enable efficient, sequential experimentation in asynchronous, agent-based supercomputing environments.
• To integrate optimal design strategies from statistics with flexible active learning approaches for improved response surface estimation.

Proposed method

• The method employs a Bayesian treed Gaussian process (B-TGP) as a surrogate model to capture nonstationary spatial behavior in the response surface.
• Predictive uncertainty from the B-TGP is used to guide the selection of new experimental runs, focusing on regions of high uncertainty.
• A fully Bayesian inference approach provides explicit, nonstationary measures of uncertainty for decision-making.
• The framework supports asynchronous, random, agent-based execution, making it suitable for large-scale supercomputing environments.
• It combines optimal design principles from statistics with active learning strategies to balance exploration and exploitation.

Experimental results

Research questions

• RQ1 How can we efficiently explore high-dimensional, nonstationary response surfaces in computationally expensive supercomputer experiments?
• RQ2 What sampling strategy minimizes the number of simulations while maximizing model accuracy and spatial coverage?
• RQ3 How can predictive uncertainty be effectively used to guide sequential experimental design in a distributed, asynchronous computing environment?
• RQ4 To what extent does the Bayesian treed Gaussian process outperform standard stationary models in capturing complex, nonstationary response surfaces?
• RQ5 How can optimal design principles be integrated with active learning for adaptive supercomputer experimentation?

Key findings

• The adaptive design framework significantly improves response surface estimation accuracy compared to fixed sparse designs, especially in nonstationary regions.
• The use of predictive uncertainty enables effective exploration of high-uncertainty regions, reducing the number of required simulations.
• The Bayesian treed Gaussian process provides explicit, nonstationary uncertainty measures that enhance decision-making in sequential design.
• The framework is robust in asynchronous, agent-based supercomputing environments, enabling scalable and distributed experimentation.
• The approach demonstrates strong performance in a real-world computational fluid dynamics simulation of a rocket booster, validating its practical utility.