[Paper Review] The Implicit Bias of Benign Overfitting

Ohad Shamir|arXiv (Cornell University)|Jan 27, 2022

Machine Learning and Algorithms4 citations

TL;DR

This paper investigates the implicit bias of standard training methods in linear models, showing that benign overfitting—perfect training fit with low generalization error—typically fails in non-well-specified linear regression due to min-norm interpolating predictors being inconsistent. In contrast, for classification, the max-margin predictor is implicitly biased toward minimizing a weighted squared hinge loss, enabling benign overfitting in linearly separable settings with arbitrary label noise up to 1/2.

ABSTRACT

The phenomenon of benign overfitting, where a predictor perfectly fits noisy training data while attaining near-optimal expected loss, has received much attention in recent years, but still remains not fully understood beyond well-specified linear regression setups. In this paper, we provide several new results on when one can or cannot expect benign overfitting to occur, for both regression and classification tasks. We consider a prototypical and rather generic data model for benign overfitting of linear predictors, where an arbitrary input distribution of some fixed dimension $k$ is concatenated with a high-dimensional distribution. For linear regression which is not necessarily well-specified, we show that the minimum-norm interpolating predictor (that standard training methods converge to) is biased towards an inconsistent solution in general, hence benign overfitting will generally not occur. Moreover, we show how this can be extended beyond standard linear regression, by an argument proving how the existence of benign overfitting on some regression problems precludes its existence on other regression problems. We then turn to classification problems, and show that the situation there is much more favorable. Specifically, we prove that the max-margin predictor (to which standard training methods are known to converge in direction) is asymptotically biased towards minimizing a weighted \emph{squared hinge loss}. This allows us to reduce the question of benign overfitting in classification to the simpler question of whether this loss is a good surrogate for the misclassification error, and use it to show benign overfitting in some new settings.

Motivation & Objective

• Understand when benign overfitting occurs in linear models beyond well-specified regression.
• Investigate the implicit bias of standard training methods (min-norm and max-margin predictors) in high-dimensional settings.
• Characterize conditions under which benign overfitting is possible in classification, even with significant label noise.
• Extend theoretical understanding of benign overfitting beyond square loss regression to classification and non-well-specified models.

Proposed method

• Proposes a prototypical data model: a k-dimensional input distribution concatenated with a high-dimensional, approximately orthogonal distribution.
• Analyzes the asymptotic behavior of the minimum-norm interpolating predictor in linear regression as dimension d and sample size m diverge with d ≫ m.
• Derives the asymptotic form of the min-norm predictor, showing it converges to a biased solution unless the data is well-specified.
• For classification, proves that the max-margin predictor asymptotically minimizes a weighted squared hinge loss under the data distribution.
• Uses the equivalence between minimizing this loss and achieving low misclassification error to establish benign overfitting in linearly separable settings.
• Applies perturbation analysis to show that high-dimensional noise components decay in influence, enabling consistent estimation under mild conditions.

Experimental results

Research questions

• RQ1 Under what conditions does benign overfitting fail in non-well-specified linear regression?
• RQ2 How does the implicit bias of the minimum-norm interpolating predictor affect generalization in high-dimensional regression?
• RQ3 Can benign overfitting occur in classification even with arbitrary label noise, and under what distributional assumptions?
• RQ4 What is the implicit bias of the max-margin predictor in high-dimensional, linearly separable settings?
• RQ5 How does the structure of the input distribution—particularly a low-dimensional component and a high-dimensional i.i.d. component—affect the feasibility of benign overfitting?

Key findings

• In non-well-specified linear regression, the minimum-norm interpolating predictor is asymptotically biased and inconsistent, precluding benign overfitting.
• The min-norm predictor converges to a solution that depends on the inverse of the expected outer product of the high-dimensional component, not the true least-squares solution.
• Benign overfitting in regression is precluded if it exists on one problem, due to structural constraints on the predictor's bias.
• For classification, the max-margin predictor is implicitly biased toward minimizing a weighted squared hinge loss under the data distribution.
• Benign overfitting occurs in classification for any label noise level up to 1/2 when the data distribution is linearly separable and satisfies symmetry and independence conditions on a distinguished direction.
• The results hold asymptotically under the condition that dimension d grows sufficiently faster than sample size m, ensuring high-dimensional noise components decay in influence.