[Paper Review] Don't Waste Your Time: Early Stopping Cross-Validation

Edward M. Bergman, Lennart Purucker+1 more|arXiv (Cornell University)|May 6, 2024

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes a simple early stopping strategy for k-fold cross-validation in automated machine learning (AutoML), where validation is halted after the first fold if a configuration underperforms the current best. The method enables model selection to converge 214% faster and evaluate 167% more configurations within one hour, while improving overall performance across 36 tabular datasets using MLP and random forest models with 3-, 5-, and 10-fold cross-validation.

ABSTRACT

State-of-the-art automated machine learning systems for tabular data often employ cross-validation; ensuring that measured performances generalize to unseen data, or that subsequent ensembling does not overfit. However, using k-fold cross-validation instead of holdout validation drastically increases the computational cost of validating a single configuration. While ensuring better generalization and, by extension, better performance, the additional cost is often prohibitive for effective model selection within a time budget. We aim to make model selection with cross-validation more effective. Therefore, we study early stopping the process of cross-validation during model selection. We investigate the impact of early stopping on random search for two algorithms, MLP and random forest, across 36 classification datasets. We further analyze the impact of the number of folds by considering 3-, 5-, and 10-folds. In addition, we investigate the impact of early stopping with Bayesian optimization instead of random search and also repeated cross-validation. Our exploratory study shows that even a simple-to-understand and easy-to-implement method consistently allows model selection to converge faster; in ~94% of all datasets, on average by ~214%. Moreover, stopping cross-validation enables model selection to explore the search space more exhaustively by considering +167% configurations on average within one hour, while also obtaining better overall performance.

Motivation & Objective

• To address the high computational cost of k-fold cross-validation in AutoML, which limits exhaustive hyperparameter search within time budgets.
• To investigate whether early stopping during cross-validation can accelerate model selection without sacrificing performance.
• To evaluate simple, easy-to-implement early stopping strategies that avoid complex hyperparameters or configuration pausing.
• To demonstrate that even basic early stopping methods can significantly improve efficiency and effectiveness in AutoML workflows.

Proposed method

• The authors implement two simple early stopping rules: 'Aggressive' (stop after first fold if performance is worse than the incumbent) and 'Forgiving' (stop after first fold if performance is worse than the incumbent by a threshold).
• These methods are applied during random search and Bayesian optimization for MLP and random forest models across 36 classification datasets from the AutoML Benchmark.
• Cross-validation is performed with 3-, 5-, and 10-fold splits, and repeated 10-fold cross-validation is also evaluated.
• The performance of early stopping is compared against standard cross-validation with no early stopping, using a fixed one-hour time budget.
• The method avoids configuration pausing and does not require complex statistical testing or hyper-hyper-parameters, minimizing implementation barriers.
• Experiments are reproducible and released with full code, documentation, and raw results on GitHub.

Figure 1: Speedup Overview Per Dataset for MLP with 10 folds: The time point of matching the best performance of No ES for each method per dataset. A marker indicates when a method reached the same or better performance than No ES . The solid black line visualizes the time saved by the fastest early

Experimental results

Research questions

• RQ1 Does early stopping during k-fold cross-validation significantly reduce time-to-convergence in AutoML model selection?
• RQ2 Can early stopping allow for more extensive exploration of the hyperparameter search space within a fixed time budget?
• RQ3 How does early stopping impact final model performance compared to full cross-validation?
• RQ4 How do different numbers of folds (3, 5, 10) affect the benefits of early stopping?
• RQ5 Can simple, easy-to-implement early stopping rules outperform standard cross-validation in practice?

Key findings

• In approximately 94% of the 36 datasets, early stopping enabled model selection to converge 214% faster on average compared to full cross-validation.
• Within a one-hour time budget, early stopping allowed the evaluation of 167% more hyperparameter configurations on average than standard cross-validation.
• The method consistently improved final model performance, indicating that early stopping does not compromise generalization despite reduced validation cost.
• The benefits were observed across different algorithms (MLP and random forest), fold counts (3, 5, 10), and optimization strategies (random search and Bayesian optimization).
• The 'Aggressive' and 'Forgiving' early stopping rules were both effective and easy to implement, with minimal overhead and no need for complex statistical testing or configuration pausing.
• The study estimates a total compute cost of ~6.15 CPU years, with 10% overhead, for running all experiments across datasets, methods, and configurations.

Figure 2: Footprint Plot for OpenML Task ID 168350 on outer fold 7: A configuration footprint, that is, a multi-dimensional scaling (MDS) embedding of the high dimensional search space for MLP to a 2-dimensional one, showing the landscape of evaluated configurations that were either evaluated (big m