[Paper Review] Testing for Overfitting

James Schmidt|arXiv (Cornell University)|May 9, 2023

Machine Learning and Algorithms4 citations

TL;DR

This paper proposes a statistical hypothesis test to detect overfitting in machine learning models using training data alone, leveraging concentration inequalities to compare empirical performance on training and holdout data. The method enables quantitative detection of overfitting without relying on holdout sets for generalization assessment, offering a rigorous alternative to heuristic early stopping.

ABSTRACT

High complexity models are notorious in machine learning for overfitting, a phenomenon in which models well represent data but fail to generalize an underlying data generating process. A typical procedure for circumventing overfitting computes empirical risk on a holdout set and halts once (or flags that/when) it begins to increase. Such practice often helps in outputting a well-generalizing model, but justification for why it works is primarily heuristic. We discuss the overfitting problem and explain why standard asymptotic and concentration results do not hold for evaluation with training data. We then proceed to introduce and argue for a hypothesis test by means of which both model performance may be evaluated using training data, and overfitting quantitatively defined and detected. We rely on said concentration bounds which guarantee that empirical means should, with high probability, approximate their true mean to conclude that they should approximate each other. We stipulate conditions under which this test is valid, describe how the test may be used for identifying overfitting, articulate a further nuance according to which distributional shift may be flagged, and highlight an alternative notion of learning which usefully captures generalization in the absence of uniform PAC guarantees.

Motivation & Objective

• To address the lack of rigorous statistical justification for using holdout sets to detect overfitting in machine learning.
• To develop a method that quantitatively defines and detects overfitting using only training data, avoiding reliance on holdout sets for generalization evaluation.
• To provide a test that remains valid even when uniform PAC learnability guarantees do not hold, enabling generalization assessment under weaker assumptions.
• To clarify the concept of overfitting by grounding it in statistical hypothesis testing rather than heuristic practices like early stopping.
• To introduce a notion of $π$-learnability as a weaker but still useful alternative to PAC learnability for practical model evaluation.

Proposed method

• The method uses Hoeffding's inequality to bound the probability that the empirical mean of a cost function deviates from its true expectation.
• It formulates a statistical hypothesis test comparing empirical risk on training data and holdout data, using concentration bounds to assess whether observed differences are statistically significant.
• The test is based on the principle that, with high probability, empirical means should approximate each other if the model is not overfitting.
• The procedure defines a precision threshold $\varepsilon$ and confidence level $\delta$, and uses Hoeffding’s inequality to compute a required sample size for reliable detection.
• The test is applied across training epochs to detect when performance on training data begins to diverge significantly from holdout performance, signaling overfitting.
• The method allows for the detection of overfitting without requiring the hypothesis class to be PAC-learnable, relying instead on distribution-specific learnability ($\pi$-learnability).

Figure 1 : Slicing the Cost Function $c:\mathcal{H}\times(\mathcal{X}\times\mathcal{Y})^{\omega}\rightarrow\mathbb{R}$

Experimental results

Research questions

• RQ1 Can overfitting be detected using only training data, without relying on holdout sets?
• RQ2 What statistical framework can rigorously define and detect overfitting in the absence of uniform PAC guarantees?
• RQ3 How can concentration inequalities be used to construct a valid hypothesis test for overfitting on training data?
• RQ4 In what way does the proposed test improve upon heuristic practices like early stopping?
• RQ5 What is the role of distribution-specific learnability ($\pi$-learnability) in enabling generalization assessment when PAC learnability does not hold?

Key findings

• The proposed hypothesis test successfully detects overfitting by comparing empirical risk on training and holdout data using concentration bounds, with statistical validity guaranteed under specified conditions.
• The method enables detection of overfitting without requiring the hypothesis class to be PAC-learnable, thus broadening applicability to real-world learning problems.
• Simulation results on synthetic data and MNIST show that the test identifies overfitting as performance on training data begins to diverge from holdout performance, even when early stopping is not applied.
• The test remains effective even when models overfit and then recover generalization performance at later epochs, demonstrating sensitivity to dynamic overfitting patterns.
• The use of Hoeffding’s inequality allows for precise control of Type I error rates through confidence level $\delta$ and precision $\varepsilon$, with sample size $m$ determining the bound's tightness.
• The framework introduces $\pi$-learnability as a meaningful alternative to PAC learnability, allowing generalization guarantees under weaker, distribution-specific assumptions.

Figure 2 : Sequences of models $\hat{y}_{\mathsf{S_{1}}},\hat{y}_{\mathsf{S_{2}}},\ldots,\hat{y}_{\mathsf{S_{m}}},\ldots$