[Paper Review] Beyond Random Split for Assessing Statistical Model Performance

Carlos Catania, J.E. Castillo Guerra+3 more|arXiv (Cornell University)|Sep 4, 2022

Neural Networks and Applications4 citations

TL;DR

This paper proposes non-random data splitting strategies—dissimilarity-based, informed-based, and clustering-based—to improve generalization error estimation in machine learning when datasets are non-representative, especially for rare or elusive examples. Using the CTU19 botnet dataset and two tree-based classifiers, the study shows that alternative splitting strategies, particularly clustering-based and informed-based, provide more realistic and pessimistic performance estimates than standard Monte Carlo splitting, especially under low-representativeness scenarios.

ABSTRACT

Even though a train/test split of the dataset randomly performed is a common practice, could not always be the best approach for estimating performance generalization under some scenarios. The fact is that the usual machine learning methodology can sometimes overestimate the generalization error when a dataset is not representative or when rare and elusive examples are a fundamental aspect of the detection problem. In the present work, we analyze strategies based on the predictors' variability to split in training and testing sets. Such strategies aim at guaranteeing the inclusion of rare or unusual examples with a minimal loss of the population's representativeness and provide a more accurate estimation about the generalization error when the dataset is not representative. Two baseline classifiers based on decision trees were used for testing the four splitting strategies considered. Both classifiers were applied on CTU19 a low-representative dataset for a network security detection problem. Preliminary results showed the importance of applying the three alternative strategies to the Monte Carlo splitting strategy in order to get a more accurate error estimation on different but feasible scenarios.

Motivation & Objective

• To address the overestimation of model generalization error when using random train/test splits on non-representative datasets.
• To evaluate the impact of different data splitting strategies on model performance estimation, particularly when rare or unusual examples are underrepresented.
• To assess whether predictor-based sampling strategies can improve the accuracy of generalization error estimation compared to standard random splitting.
• To provide a framework for evaluating model robustness under varying levels of data representativeness, especially in network security detection tasks.
• To demonstrate that alternative splitting strategies yield more realistic performance estimates than Monte Carlo splitting in low-representative scenarios.

Proposed method

• Employs four data splitting strategies: Monte Carlo (random split), dissimilarity-based (maximizing distance between training and test samples), informed-based (using group-level information to isolate test sets), and clustering-based (using k-means to partition predictor space).
• Applies two baseline decision tree classifiers (CART and C5.0) to the CTU19 dataset, a low-representative network security dataset for botnet detection.
• Uses the algorithm from [3] to quantify the degree of overlap between training and testing sets, measuring representativeness differences.
• Evaluates model performance using standard metrics: Balanced Accuracy, F1, Sensitivity, and Specificity across all splitting strategies.
• Conducts multiple resamples (Monte Carlo-style) for each strategy to assess stability and variability in performance estimates.
• Compares median and interquartile range of performance metrics across strategies to assess robustness and estimation accuracy.

Experimental results

Research questions

• RQ1 How does the choice of data splitting strategy affect the estimation of model generalization error in non-representative datasets?
• RQ2 Can predictor-based sampling strategies (dissimilarity, clustering, informed grouping) improve the accuracy of performance estimation compared to random splitting?
• RQ3 What is the impact of varying representativeness between training and testing sets on model performance metrics like Balanced Accuracy and F1?
• RQ4 How do different splitting strategies reflect the model’s robustness under realistic but challenging data distribution shifts?
• RQ5 To what extent do alternative splitting strategies provide more pessimistic and thus more realistic performance estimates than standard Monte Carlo splitting?

Key findings

• The Monte Carlo splitting strategy, while standard, overestimates model performance (median Balanced Accuracy = 0.91) due to high similarity between training and testing sets.
• The dissimilarity-based strategy increases estimation error (median Balanced Accuracy = 0.75) due to poor initial set selection, indicating instability in small datasets.
• The informed-based splitting strategy shows a wider performance range (Balanced Accuracy from 0.75 to 0.91), revealing that 0.75 may represent a worst-case scenario under low representativeness.
• The clustering-based strategy yields the most pessimistic estimate (median Balanced Accuracy = 0.82), with higher F1 (0.93) and increased variation, indicating robustness testing under extreme distribution shifts.
• The clustering-based and informed-based strategies provide more realistic performance estimates than Monte Carlo, especially for real-world deployment in non-representative data scenarios.
• All strategies confirmed that small differences in training/test set representativeness significantly affect performance metrics, especially F1 and Sensitivity, highlighting the need for careful splitting in model evaluation.