[Paper Review] More Data Can Hurt for Linear Regression: Sample-wise Double Descent

Preetum Nakkiran|arXiv (Cornell University)|Dec 16, 2019

Random Matrices and Applications12 references42 citations

TL;DR

The paper analyzes overparameterized linear regression with isotropic Gaussian covariates and shows that test risk can be non-monotonic in the number of samples, peaking near n = d due to a bias-variance tradeoff.

ABSTRACT

In this expository note we describe a surprising phenomenon in overparameterized linear regression, where the dimension exceeds the number of samples: there is a regime where the test risk of the estimator found by gradient descent increases with additional samples. In other words, more data actually hurts the estimator. This behavior is implicit in a recent line of theoretical works analyzing "double-descent" phenomenon in linear models. In this note, we isolate and understand this behavior in an extremely simple setting: linear regression with isotropic Gaussian covariates. In particular, this occurs due to an unconventional type of bias-variance tradeoff in the overparameterized regime: the bias decreases with more samples, but variance increases.

Motivation & Objective

• Motivate understanding of non-monotonic test risk in overparameterized linear models.
• Isolate the sample-size regime where adding data hurts performance.
• Provide intuition and approximate bias-variance expressions to explain the phenomenon.

Proposed method

• Study minimum-norm ridgeless regression estimator, equivalently gradient descent on least squares.
• Decompose excess risk into bias and variance components and derive approximate expressions B_n and V_n.
• Analyze conditioning of the data matrix X and its effect on the trace term Tr((XX^T)^{-1}).
• Use isotropic Gaussian covariates X ~ N(0,I_d) and y = ⟨x,β⟩ + η with ||β||_2 ≤ 1.
• Provide claims giving closed-form approximations in the n ≤ d regime and reference underparameterized results for n > d.

Experimental results

Research questions

• RQ1 How does the test risk of the minimum-norm interpolating estimator behave as a function of the number of samples n with fixed dimension d?
• RQ2 What are the bias and variance contributions to excess risk in the overparameterized regime (n ≤ d)?
• RQ3 Why does the data matrix X become poorly conditioned around the critical regime n ≈ d, leading to heightened variance?
• RQ4 How does adding a single sample affect the trace term Tr((XX^T)^{-1}) and overall risk?
• RQ5 Do the theoretical approximations align with empirical observations for finite d (e.g., d = 1000)?

Key findings

• Test risk is non-monotonic in n; it decreases first, peaks at n = d, then decreases again as n increases beyond d.
• In the overparameterized regime, bias B_n decreases with n while variance V_n increases and dominates near the critical point.
• Approximate excess risk for γ = n/d < 1 is E[R̄(β̂)] ≈ (1 − γ)||β||^2 + σ^2 γ/(1−γ).
• The peak in risk is tied to poor conditioning of X when n ≈ d, causing the noise term X^†η to inflate with high norm.
• The trace term in variance satisfies Tr((XX^T)^{-1}) → γ/(1−γ) as d grows with n = γd, explaining the variance surge.
• Exact finite-sample bias and variance expressions are provided for the n ≤ d regime.