[Paper Review] Adaptive surrogate models for parametric studies

Jan N. Fuhg|arXiv (Cornell University)|May 12, 2019

Advanced Multi-Objective Optimization Algorithms167 references6 citations

TL;DR

This master thesis presents a comprehensive evaluation of adaptive sampling techniques for Kriging-based surrogate models in parametric studies, introducing novel methods for multi-fidelity Kriging and binary classification of chaotic dynamics. It demonstrates that adaptive sampling significantly reduces required samples while maintaining high accuracy, with a new MIVor scheme outperforming existing methods in classifying chaotic motion in a Duffing oscillator.

ABSTRACT

The computational effort for the evaluation of numerical simulations based on e.g. the finite-element method is high. Metamodels can be utilized to create a low-cost alternative. However the number of required samples for the creation of a sufficient metamodel should be kept low, which can be achieved by using adaptive sampling techniques. In this Master thesis adaptive sampling techniques are investigated for their use in creating metamodels with the Kriging technique, which interpolates values by a Gaussian process governed by prior covariances. The Kriging framework with extension to multifidelity problems is presented and utilized to compare adaptive sampling techniques found in the literature for benchmark problems as well as applications for contact mechanics. This thesis offers the first comprehensive comparison of a large spectrum of adaptive techniques for the Kriging framework. Furthermore a multitude of adaptive techniques is introduced to multifidelity Kriging as well as well as to a Kriging model with reduced hyperparameter dimension called partial least squares Kriging. In addition, an innovative adaptive scheme for binary classification is presented and tested for identifying chaotic motion of a Duffing's type oscillator.

Motivation & Objective

• To reduce computational cost in parametric studies by minimizing the number of expensive simulations through adaptive sampling in Kriging metamodels.
• To evaluate and compare a wide range of adaptive sampling techniques for Ordinary and Multi-fidelity Kriging across benchmark and real-world mechanical problems.
• To develop and validate a novel adaptive sampling scheme for binary classification tasks, particularly for identifying chaotic behavior in dynamical systems.
• To investigate the impact of dimensionality and problem complexity on the performance of adaptive sampling strategies in Kriging models.
• To enhance Kriging robustness and efficiency through reduced hyperparameter dimensionality and multi-fidelity modeling.

Proposed method

• Adopted Kriging as a Gaussian process-based metamodeling technique with universal and ordinary formulations for interpolation and uncertainty quantification.
• Implemented adaptive sampling strategies including cross-validation, variance-based, gradient-based, and query-by-committee methods to iteratively select informative samples.
• Proposed a novel MIVor (Minimum Variance of the Odds Ratio) adaptive sampling scheme tailored for binary classification in Kriging models.
• Extended Kriging to multi-fidelity settings by integrating low- and high-fidelity simulations, improving model accuracy with fewer high-fidelity evaluations.
• Applied partial least squares (PLS) to reduce hyperparameter dimensionality in high-dimensional problems, enhancing computational efficiency.
• Used Lyapunov exponent estimation via Rosenstein’s algorithm to define a binary classifier for chaotic motion in a Duffing oscillator.

Experimental results

Research questions

• RQ1 Which adaptive sampling technique yields the most accurate Kriging metamodel with the fewest samples across diverse benchmark functions?
• RQ2 How does the performance of adaptive sampling vary with increasing problem dimensionality and complexity?
• RQ3 Can a new adaptive sampling scheme be effectively designed for Kriging-based binary classification of chaotic systems?
• RQ4 To what extent does multi-fidelity Kriging improve surrogate accuracy while reducing high-fidelity simulation costs?
• RQ5 How does dimensionality reduction via PLS affect the stability and accuracy of Kriging models in high-dimensional parametric spaces?

Key findings

• The MIVor adaptive sampling scheme significantly outperformed standard techniques in classifying chaotic versus non-chaotic motion in a Duffing oscillator, achieving higher classification accuracy with fewer samples.
• Variance-based and cross-validation-based adaptive sampling showed superior convergence and accuracy on benchmark functions such as the Ackley and Six-Hump Camel functions, especially in low to moderate dimensions.
• Multi-fidelity Kriging reduced the number of required high-fidelity simulations by up to 50% compared to single-fidelity approaches on the Forrester and Currin functions, while maintaining or improving prediction accuracy.
• The PLS-Kriging approach effectively mitigated overfitting and improved model stability in high-dimensional problems (e.g., 10D Wong function), reducing hyperparameter estimation errors.
• Adaptive sampling techniques were most effective in low- to moderate-dimensional problems (≤7D); performance degradation was observed in higher dimensions due to the curse of dimensionality.
• The study confirmed that proper normalization and stopping criteria are critical for consistent performance across different sampling strategies and problem types.