[Paper Review] Model predictivity assessment: incremental test-set selection and accuracy evaluation

Elias Fekhari, Bertrand Iooss+3 more|arXiv (Cornell University)|Jul 8, 2022

Industrial Vision Systems and Defect Detection63 references4 citations

TL;DR

This paper proposes an incremental test-set selection method for machine learning model predictivity assessment using Maximum Mean Discrepancy (MMD)-based experimental design and weighted error estimation. By optimally selecting test points to maximize coverage of the input space and applying MMD-driven weights to prediction errors, the approach achieves superior accuracy in error estimation with significantly reduced computational cost compared to cross-validation, as validated on an industrial nuclear safety simulation case study.

ABSTRACT

Unbiased assessment of the predictivity of models learnt by supervised machine-learning methods requires knowledge of the learned function over a reserved test set (not used by the learning algorithm). The quality of the assessment depends, naturally, on the properties of the test set and on the error statistic used to estimate the prediction error. In this work we tackle both issues, proposing a new predictivity criterion that carefully weights the individual observed errors to obtain a global error estimate, and using incremental experimental design methods to "optimally" select the test points on which the criterion is computed. Several incremental constructions are studied, including greedy-packing (coffee-house design), support points and kernel herding techniques. Our results show that the incremental and weighted versions of the latter two, based on Maximum Mean Discrepancy concepts, yield superior performance. An industrial test case provided by the historical French electricity supplier (EDF) illustrates the practical relevance of the methodology, indicating that it is an efficient alternative to expensive cross-validation techniques.

Motivation & Objective

• To improve the reliability of model predictivity assessment by selecting test sets that are representative of the input space, avoiding bias from random or poorly distributed test points.
• To address the challenge of limited data in industrial applications where expensive simulations or experiments restrict the size of available datasets.
• To develop a computationally efficient alternative to cross-validation for evaluating machine learning models, especially in high-fidelity simulation contexts such as nuclear reactor safety analysis.
• To integrate MMD-based experimental design with weighted error statistics to enhance the sensitivity and accuracy of prediction error estimation.
• To demonstrate the practical effectiveness of the method on a real-world industrial case study involving thermal-hydraulic simulations in pressurized water reactors.

Proposed method

• Uses incremental experimental design techniques—specifically greedy-packing, support points, and kernel herding—to select test points that maximize space-filling properties in the input domain.
• Applies a novel weighted error criterion based on Maximum Mean Discrepancy (MMD) to reweight individual prediction residuals, improving global error estimation accuracy.
• Employs kernel embedding and potential functions to compute MMD between the training and test distributions, enabling principled comparison of probability measures.
• Utilizes the reproducing kernel Hilbert space (RKHS) framework to define the MMD as a norm of the difference between kernel embeddings of the training and test measures.
• Derives analytical expressions for MMD potentials under Matérn and energy distance kernels to enable efficient computation without Monte Carlo approximation.
• Combines the MMD-based test set selection with a weighted mean-squared error estimator to produce a robust, unbiased prediction error estimate.

Experimental results

Research questions

• RQ1 Can incremental MMD-based test-set selection outperform random or heuristic test-set selection in estimating model prediction error?
• RQ2 How does the use of MMD-weighted residuals improve the accuracy and reliability of model predictivity assessment compared to standard error metrics?
• RQ3 To what extent can the proposed method reduce computational cost relative to cross-validation while maintaining or improving estimation accuracy?
• RQ4 How do different incremental design strategies—greedy-packing, support points, kernel herding—compare in terms of test-set representativeness and error estimation performance?
• RQ5 Does the proposed method maintain robustness and accuracy when applied to real industrial datasets with limited size and high computational cost?

Key findings

• Kernel herding and support point methods based on MMD outperform greedy-packing and random selection in terms of error estimation accuracy and stability.
• The MMD-weighted error estimator provides a more reliable and less biased estimate of the true prediction error than standard mean-squared error on unweighted residuals.
• The proposed method achieves comparable or better accuracy than 10-fold cross-validation while reducing computational cost by avoiding repeated model retraining.
• Analytical computation of MMD potentials for Matérn kernels enables fast and accurate evaluation of discrepancy without Monte Carlo sampling.
• In the industrial case study involving thermal-hydraulic simulations, the method successfully identified model limitations with a small, well-chosen test set, demonstrating practical utility.
• The incremental selection process allows for adaptive refinement of the test set when additional computational resources become available, enhancing model validation flexibility.