[Paper Review] Constructing Confidence Intervals for 'the' Generalization Error -- a Comprehensive Benchmark Study

Hannah Schulz-Kümpel, Sebastian Fischer+4 more|arXiv (Cornell University)|Sep 27, 2024

Scientific Measurement and Uncertainty EvaluationDecision Sciences3 citations

TL;DR

This paper presents a comprehensive benchmark of 13 model-agnostic methods for constructing confidence intervals (CIs) for the generalization error in machine learning, using 18 tabular datasets across 4 learning algorithms and 8 loss functions. It evaluates coverage frequency, width, and runtime, recommending Nested Cross-Validation with 25 outer repetitions and K=5 for small data (n ≤ 100), and Corrected Resampled-t with ratio 0.9 and 25 repetitions for larger datasets.

ABSTRACT

When assessing the quality of prediction models in machine learning, confidence intervals (CIs) for the generalization error, which measures predictive performance, are a crucial tool. Luckily, there exist many methods for computing such CIs and new promising approaches are continuously being proposed. Typically, these methods combine various resampling procedures, most popular among them cross-validation and bootstrapping, with different variance estimation techniques. Unfortunately, however, there is currently no consensus on when any of these combinations may be most reliably employed and how they generally compare. In this work, we conduct a large-scale study comparing CIs for the generalization error, the first one of such size, where we empirically evaluate 13 different CI methods on a total of 19 tabular regression and classification problems, using seven different inducers and a total of eight loss functions. We give an overview of the methodological foundations and inherent challenges of constructing CIs for the generalization error and provide a concise review of all 13 methods in a unified framework. Finally, the CI methods are evaluated in terms of their relative coverage frequency, width, and runtime. Based on these findings, we can identify a subset of methods that we would recommend. We also publish the datasets as a benchmarking suite on OpenML and our code on GitHub to serve as a basis for further studies.

Motivation & Objective

• To address the lack of consensus on which resampling and variance estimation methods reliably produce accurate confidence intervals (CIs) for the generalization error in machine learning.
• To empirically compare 13 existing, model-agnostic methods for constructing CIs using a large-scale benchmark across diverse tabular regression and classification problems.
• To evaluate the performance of these methods in terms of coverage frequency, CI width, computational cost, and stability across different data sizes, models, and loss functions.
• To provide actionable recommendations for practitioners based on empirical results, supported by open-source code and datasets on OpenML and GitHub.
• To establish a foundation for future methodological evaluations by offering a standardized benchmark suite and highlighting the importance of empirical validation alongside theoretical analysis.

Proposed method

• The study evaluates 13 existing, model-agnostic methods for constructing CIs for the generalization error, including variants of cross-validation, bootstrapping, and variance estimation techniques.
• Each method combines resampling procedures (e.g., K-fold cross-validation, repeated cross-validation, bootstrapping) with different variance estimation strategies (e.g., jackknife, variance ratio, t-statistic corrections).
• The benchmark uses 18 tabular datasets—17 specifically generated for this study—covering 4 supervised learning algorithms (inducers) and 8 loss functions, across both classification and regression tasks.
• Performance is assessed using three primary metrics: relative coverage frequency (how often the true generalization error lies within the CI), average width (precision), and runtime (computational cost).
• The study employs a unified framework to describe and compare all 13 methods, enabling consistent evaluation and interpretation across diverse approaches.
• All datasets are published on OpenML, and the code is released on GitHub to support reproducibility and future benchmarking.

Experimental results

Research questions

• RQ1 Which combinations of resampling and variance estimation methods yield the most reliable coverage frequency for confidence intervals of the generalization error?
• RQ2 How do the widths of the confidence intervals vary across methods, and what is the trade-off between coverage accuracy and precision?
• RQ3 What is the computational cost of each method, and how does it scale with data size and model complexity?
• RQ4 How do the performance characteristics of the methods differ between small data (n ≤ 100) and larger data (n > 100)?
• RQ5 Which methods demonstrate robustness and stability across diverse data-generating processes, learning algorithms, and loss functions?

Key findings

• For small datasets (n ≤ 100), Nested Cross-Validation with at least 25 outer repetitions and K=5, or Conservative-Z with 25 outer repetitions and K≥10, provide the most reliable coverage with acceptable width.
• For larger datasets, the Corrected Resampled-t method with a ratio of 0.9 and at least 25 repetitions achieves the best balance of coverage accuracy and width.
• Nested Cross-Validation with 3 outer repetitions and K=5, or Conservative-Z with 10 outer repetitions and K=5, are also recommended for larger data when slightly wider CIs are acceptable.
• The study identifies that coverage frequency is highly sensitive to method configuration, with some methods showing significant over- or under-coverage depending on data size and model type.
• Computational cost varies widely: methods involving repeated model fitting (e.g., repeated cross-validation) are significantly slower than single-pass or variance-estimation-based approaches.
• The benchmark reveals that no single method universally outperforms others across all metrics, but several methods consistently rank among the top performers in terms of coverage and width balance.