[Paper Review] The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime

Andrea Montanari, Feng Ruan+2 more|arXiv (Cornell University)|Nov 5, 2019

Neural Networks and ApplicationsComputer Science56 references90 citations

TL;DR

This paper derives exact high-dimensional asymptotics for the generalization error of max-margin linear classifiers in overparametrized settings, identifies conditions for benign overfitting, and analyzes random features models.

ABSTRACT

Modern machine learning classifiers often exhibit vanishing classification error on the training set. They achieve this by learning nonlinear representations of the inputs that maps the data into linearly separable classes. Motivated by these phenomena, we revisit high-dimensional maximum margin classification for linearly separable data. We consider a stylized setting in which data $(y_i,{\boldsymbol x}_i)$, $i\le n$ are i.i.d. with ${\boldsymbol x}_i\sim\mathsf{N}({\boldsymbol 0},{\boldsymbol Σ})$ a $p$-dimensional Gaussian feature vector, and $y_i \in\{+1,-1\}$ a label whose distribution depends on a linear combination of the covariates $\langle {\boldsymbol θ}_*,{\boldsymbol x}_i angle$. While the Gaussian model might appear extremely simplistic, universality arguments can be used to show that the results derived in this setting also apply to the output of certain nonlinear featurization maps. We consider the proportional asymptotics $n,p o\infty$ with $p/n o ψ$, and derive exact expressions for the limiting generalization error. We use this theory to derive two results of independent interest: $(i)$ Sufficient conditions on $({\boldsymbol Σ},{\boldsymbol θ}_*)$ for `benign overfitting' that parallel previously derived conditions in the case of linear regression; $(ii)$ An asymptotically exact expression for the generalization error when max-margin classification is used in conjunction with feature vectors produced by random one-layer neural networks.

Motivation & Objective

• Motivate study of max-margin classifiers in high-dimensional, overparametrized regimes with vanishing training error.
• Characterize when these classifiers generalize well (benign overfitting) under Gaussian feature models.
• Provide explicit asymptotic formulas for generalization error and interpolation thresholds.
• Extend results to random features models and wide neural/network-inspired featurizations.
• Offer conditions on covariance structure and signal alignment that govern generalization behavior.

Proposed method

• Assume i.i.d. data with Gaussian features x_i ~ N(0, Σ) and labels y_i distributed via f(⟨θ*, x_i⟩).
• Adopt proportional asymptotics with n, p → ∞ and p/n → ψ.
• Derive limiting generalization error Err*(μ, ψ) for the max-margin classifier through a Gaussian-equivalent model and universality arguments.
• Characterize the interpolation threshold ψ*(μ) where positive margin becomes possible.
• Analyze a random features model where features are outputs of a single random hidden layer and apply universality to obtain exact asymptotics.
• Use Gordon’s Gaussian comparison framework to reduce the problem to a nearly separable convex-concave form and extract nonlinear equation systems.

Experimental results

Research questions

• RQ1 What are the limiting generalization errors of max-margin linear classifiers in high-dimensional, overparametrized settings under Gaussian features?
• RQ2 What are the sufficient and necessary conditions on Σ and θ* that yield benign overfitting for max-margin classification?
• RQ3 How does the interpolation threshold (minimum p/n for positive margin) depend on data covariance and signal structure?
• RQ4 Do the asymptotic results extend to random features models and wide neural network regimes?
• RQ5 Can the Gaussian-equivalent approach yield exact predictions for margin and error beyond ridge regression?

Key findings

• The margin and prediction error converge to non-random limits κ*(μ, ψ) and Err*(μ, ψ) in probability as n → ∞.
• Benign overfitting occurs under certain spectral and alignment conditions on Σ and θ*, mirroring results known for linear regression.
• Overparameterization (large ψ) is necessary for max-margin classifiers to achieve near Bayes error in the studied high-dimensional regime.
• In random features models, the test error decreases with increasing width p and becomes minimal in the large overparametrization limit p/n ≫ 1.
• The analysis provides explicit bias-like and variance-like terms B_n(λ) and V_n(λ) guiding when excess error is small, and offers ε-consistency results for suitable parameter choices.