[Paper Review] Optimal Data Split Methodology for Model Validation

Rebecca Morrison, Corey Bryant+3 more|arXiv (Cornell University)|Aug 30, 2011

Probabilistic and Robust Engineering Design9 references5 citations

TL;DR

This paper proposes a systematic, optimization-based data splitting methodology for model validation that selects the most challenging calibration and validation sets to rigorously test a model's predictive capability for a specific quantity of interest (QoI). By evaluating all possible data partitions under size constraints and selecting the split that maximizes validation challenge while ensuring calibration adequacy, the method identifies whether a model can reliably predict unobserved scenarios—demonstrated by invalidating a data reduction model for an ICCD camera due to poor QoI prediction despite good data reproduction.

ABSTRACT

The decision to incorporate cross-validation into validation processes of mathematical models raises an immediate question - how should one partition the data into calibration and validation sets? We answer this question systematically: we present an algorithm to find the optimal partition of the data subject to certain constraints. While doing this, we address two critical issues: 1) that the model be evaluated with respect to predictions of a given quantity of interest and its ability to reproduce the data, and 2) that the model be highly challenged by the validation set, assuming it is properly informed by the calibration set. This framework also relies on the interaction between the experimentalist and/or modeler, who understand the physical system and the limitations of the model; the decision-maker, who understands and can quantify the cost of model failure; and the computational scientists, who strive to determine if the model satisfies both the modeler's and decision maker's requirements. We also note that our framework is quite general, and may be applied to a wide range of problems. Here, we illustrate it through a specific example involving a data reduction model for an ICCD camera from a shock-tube experiment located at the NASA Ames Research Center (ARC).

Motivation & Objective

• To address the lack of systematic, quantitative criteria for partitioning data into calibration and validation sets in model validation.
• To ensure models are both well-informed by calibration data and rigorously challenged by validation data, particularly for predicting unobservable quantities of interest (QoI).
• To develop a framework that avoids subjective or arbitrary data splits, especially in cases with limited or legacy experimental data.
• To integrate modeler, experimentalist, and decision-maker insights with computational validation to assess model credibility for predictive use.
• To provide a generalizable methodology applicable across scientific domains where predictive modeling is critical, such as nuclear stockpile maintenance or aerospace reentry simulations.

Proposed method

• The method evaluates all possible disjoint partitions of the data into calibration and validation sets of fixed sizes, constrained by a chosen calibration set size (e.g., 7 out of 11 gate widths).
• For each partition, Bayesian inference is applied using uniform priors to calibrate model parameters and compute posterior distributions.
• Two metrics are computed for each split: $M_D$, measuring the model's ability to reproduce calibration data (relative error tolerance $M^*_D = 0.2$), and $M_Q$, measuring the model's predictive performance on the QoI.
• The optimal split $s^*$ is selected as the one that maximizes $M_Q$ among all splits satisfying $M_D(s_k) < M^*_D$, ensuring the model is both calibrated and most challenged.
• The framework uses a validation pyramid concept to align model predictions with real-world QoI, ensuring the metric is directly relevant to the prediction task.
• The approach is applied to an ICCD camera data reduction model from a shock-tube experiment, using 11 gate widths as data points grouped by $Δ t$.

Experimental results

Research questions

• RQ1 What is the optimal way to partition data into calibration and validation sets to maximize the rigor of model validation?
• RQ2 How can a model be systematically challenged by the validation set while still being adequately informed by the calibration set?
• RQ3 To what extent can a model’s predictive capability for a quantity of interest be assessed when experimental data for that QoI is unavailable?
• RQ4 How can the interaction between modelers, experimentalists, and decision-makers be formalized within a computational validation framework?
• RQ5 Can a systematic, data-driven approach replace subjective or arbitrary data splitting in model validation?

Key findings

• The optimal data split $s^*$ was identified as the one where the calibration set excluded the four lowest gate widths (0.5, 0.6, 0.7, 0.8 μs), making it the most challenging for QoI prediction.
• All 330 possible data partitions were evaluated, and the model failed the validation criterion because $M_Q(s^*) > M^*_Q$, indicating it could not reliably predict the QoI.
• The model was able to reproduce calibration data with an average relative error below the 20% tolerance threshold ($M^*_D = 0.2$), confirming adequate calibration.
• The results formed four distinct groups in the analysis, reflecting increasing validation challenge based on which low-$\Delta t$ data points were excluded from calibration.
• The grouping of splits suggests potential for reducing computational cost via representative sampling or mutual information-based data clustering in future work.
• The framework successfully invalidated the ICCD camera data reduction model, demonstrating its ability to detect predictive failure even when data reproduction appears acceptable.