[Paper Review] Benign overfitting in ridge regression

Alexander Tsigler, Peter L. Bartlett|arXiv (Cornell University)|Sep 29, 2020

Sparse and Compressive Sensing Techniques16 references84 citations

TL;DR

The paper generalizes previous work on benign overfitting by eliminating independence assumptions, providing sharp nonasymptotic bounds for bias and variance in ridge regression under overparameterization, and showing conditions under which negative regularization can be optimal.

ABSTRACT

In many modern applications of deep learning the neural network has many more parameters than the data points used for its training. Motivated by those practices, a large body of recent theoretical research has been devoted to studying overparameterized models. One of the central phenomena in this regime is the ability of the model to interpolate noisy data, but still have test error lower than the amount of noise in that data. arXiv:1906.11300 characterized for which covariance structure of the data such a phenomenon can happen in linear regression if one considers the interpolating solution with minimum $\ell_2$-norm and the data has independent components: they gave a sharp bound on the variance term and showed that it can be small if and only if the data covariance has high effective rank in a subspace of small co-dimension. We strengthen and complete their results by eliminating the independence assumption and providing sharp bounds for the bias term. Thus, our results apply in a much more general setting than those of arXiv:1906.11300, e.g., kernel regression, and not only characterize how the noise is damped but also which part of the true signal is learned. Moreover, we extend the result to the setting of ridge regression, which allows us to explain another interesting phenomenon: we give general sufficient conditions under which the optimal regularization is negative.

Motivation & Objective

• Motivate understanding of why interpolating models can generalize in overparameterized settings.
• Generalize prior results to ridge regression and non-independent data components.
• Provide sharp, nonasymptotic bias and variance bounds using eigen-direction separation.
• Introduce and analyze the key matrix A_k and its condition number as central to the bounds.
• Explore conditions under which negative regularization can be optimal.

Proposed method

• Set up ridge regression in an overparameterized regime with p>n and zero-mean sub-Gaussian covariates.
• Decompose excess risk into bias B and variance V terms and express them via A, X, and the covariance spectrum.
• Introduce and leverage the eigen-direction separation: split data into first k and tail k:∞ components with A_k = X_{k:
  ∞} X_{k:∞}^{ op} +  I_n.
• Provide nonasymptotic bounds for B and V under CondNum(k,δ,L) and NoncritReg(k,γ) assumptions, with k^* as an effective switch point.
• Extend analysis to ridge regression (λ>0) and discuss conditions for negative regularization to be optimal.
• Discuss relation to prior work and discuss the sufficiency of sub-Gaussian tails via Section 5 and Section 6.

Experimental results

Research questions

• RQ1 Under what spectral conditions on the data covariance can an interpolating/overparameterized estimator achieve low generalization error?
• RQ2 How can we bound the bias and variance terms in ridge regression without independence assumptions?
• RQ3 What role does the separation of the first k eigen-directions play in achieving benign overfitting?
• RQ4 Can negative regularization be optimal for certain tail spectra and what are the sufficient conditions?
• RQ5 How does the tail behavior of the covariance influence the optimal regularization in ridge regression?

Key findings

• The bias term bound aligns with a decomposition into a high-dimensional tail part and a low-dimensional head part, showing how tail energy contributes to error.
• The variance bound generalizes Bartlett et al. by using CondNum on A_k instead of independence, yielding sharp nonasymptotic results.
• For ridge regression, the results extend to λ>0 and give conditions under which negative regularization can be optimal.
• The analysis demonstrates that benign overfitting can occur under a broader condition on the tail of the covariance, reliant on the condition number of A_k rather than independence.
• The paper provides and analyzes a central object A_k that governs both bias and variance through the tail of eigenvalues and the ridge parameter λ.
• It establishes that negative regularization can improve excess risk under certain tail and noise-energy conditions (Section 8).