[Paper Review] Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle

Rylan Schaeffer, Mikail Khona+6 more|arXiv (Cornell University)|Mar 24, 2023

Neural Networks and Applications4 citations

TL;DR

This paper demystifies double descent in machine learning by identifying three key factors—small singular values in training features, test feature variation in trailing singular modes, and non-zero residual error in the optimal model—that jointly enable the phenomenon. Using linear regression analysis and ablation studies on synthetic and real-world data, the authors show double descent vanishes when any factor is removed, offering a unified explanation for its emergence in both linear and nonlinear models like deep neural networks.

ABSTRACT

Double descent is a surprising phenomenon in machine learning, in which as the number of model parameters grows relative to the number of data, test error drops as models grow ever larger into the highly overparameterized (data undersampled) regime. This drop in test error flies against classical learning theory on overfitting and has arguably underpinned the success of large models in machine learning. This non-monotonic behavior of test loss depends on the number of data, the dimensionality of the data and the number of model parameters. Here, we briefly describe double descent, then provide an explanation of why double descent occurs in an informal and approachable manner, requiring only familiarity with linear algebra and introductory probability. We provide visual intuition using polynomial regression, then mathematically analyze double descent with ordinary linear regression and identify three interpretable factors that, when simultaneously all present, together create double descent. We demonstrate that double descent occurs on real data when using ordinary linear regression, then demonstrate that double descent does not occur when any of the three factors are ablated. We use this understanding to shed light on recent observations in nonlinear models concerning superposition and double descent. Code is publicly available.

Motivation & Objective

• To explain why double descent occurs in machine learning without relying on advanced mathematical tools like random matrix theory.
• To identify the three interpretable, interdependent factors responsible for double descent in linear models.
• To validate these factors empirically using ablation studies on both synthetic and real-world tabular datasets.
• To extend the insights from linear models to nonlinear deep learning systems, particularly in understanding generalization and memorization.
• To clarify misconceptions about noise and overfitting in double descent, showing deterministic residual errors suffice.

Proposed method

• Using polynomial regression with Legendre polynomials as basis functions to provide visual intuition for double descent behavior.
• Formalizing double descent in ordinary linear regression via analysis of the singular value decomposition (SVD) of the training feature matrix X.
• Identifying three critical conditions: (1) small singular values in X, (2) test features varying in trailing singular modes of X, and (3) non-zero residual error in the optimal model.
• Conducting ablation experiments by removing each of the three factors individually in both synthetic and real-world datasets to test their necessity.
• Projecting deep neural network activations into a feature space equivalent to linear regression, enabling application of the same framework to nonlinear models.
• Using a student-teacher framework to generate synthetic data with controlled properties, enabling causal validation of the three factors.

Experimental results

Research questions

• RQ1 What are the three specific, interpretable conditions that must all be present for double descent to occur in linear models?
• RQ2 Why does double descent emerge in overparameterized models despite classical theory predicting overfitting?
• RQ3 Does double descent persist when any of the three identified factors are removed, and what does this imply about their necessity?
• RQ4 How can insights from linear models explain double descent in nonlinear deep neural networks?
• RQ5 Is label noise or stochasticity required for double descent, or can it occur in deterministic settings?

Key findings

• Double descent in ordinary linear regression occurs only when all three factors—small singular values in X, test feature variation in trailing singular modes, and non-zero residual error—are simultaneously present.
• Ablating any one of the three factors completely eliminates double descent in both synthetic and real-world datasets, confirming their necessity.
• Double descent was observed on real tabular datasets including California Housing, Diabetes, and WHO Life Expectancy when all three factors were present.
• The phenomenon persists in nonlinear models because the effective feature space of wide neural networks can be mapped to linear regression, allowing the same three factors to govern generalization behavior.
• Double descent can occur without label noise; deterministic residual errors from model class mismatch are sufficient to produce the non-monotonic test error curve.
• The study clarifies that 'data point features' (related to XX^T) and 'generalizing features' (related to X^T X) are not mutually exclusive, and their shared spectral structure underpins the double descent behavior.