[Paper Review] Classification vs regression in overparameterized regimes: Does the loss function matter?

V. Sai Muthukumar, Adhyyan Narang+4 more|arXiv (Cornell University)|May 16, 2020

Face and Expression Recognition59 references46 citations

TL;DR

The paper shows that in highly overparameterized linear models with Gaussian features, training losses (hinge, logistic, square) yield identical interpolating predictors, while test losses differ: 0-1 loss can generalize well for classification even when square loss generalizes poorly for regression.

ABSTRACT

We compare classification and regression tasks in an overparameterized linear model with Gaussian features. On the one hand, we show that with sufficient overparameterization all training points are support vectors: solutions obtained by least-squares minimum-norm interpolation, typically used for regression, are identical to those produced by the hard-margin support vector machine (SVM) that minimizes the hinge loss, typically used for training classifiers. On the other hand, we show that there exist regimes where these interpolating solutions generalize well when evaluated by the 0-1 test loss function, but do not generalize if evaluated by the square loss function, i.e. they approach the null risk. Our results demonstrate the very different roles and properties of loss functions used at the training phase (optimization) and the testing phase (generalization).

Motivation & Objective

• Motivate and analyze differences between classification and regression under overparameterization using a linear model with Gaussian features.
• Show that training with hinge/logistic/square losses become equivalent to minimum-norm interpolation under sufficient overparameterization.
• Investigate how the choice of test loss (0-1 vs. square) affects generalization, highlighting regimes where classification generalizes well while regression does not.
• Provide non-asymptotic analysis linking SVM and minimum-ℓ2-norm interpolators for classification tasks.

Proposed method

• Model setup with an overparameterized linear predictor and Gaussian features.
• Define interpolation solutions for binary labels (minimum-ℓ2-norm) and real-valued outputs (minimum-ℓ2-norm).
• Show equivalence between hard-margin SVM and minimum-ℓ2-norm interpolation under sufficient effective overparameterization (Theorem 11).
• Extend Fourier and signal-processing interpretation to analyze when interpolating solutions generalize for classification versus regression (Theorem 13).
• Analyze ensembles (isotropic and bi-level) to study the impact of overparameterization and feature spectra on generalization.

Experimental results

Research questions

• RQ1 Does overparameterization make training losses (hinge, logistic, square) lead to the same interpolating predictor?
• RQ2 In overparameterized regimes, how does test loss (0-1 vs. squared loss) affect generalization for classification vs. regression?
• RQ3 What are the conditions under which the hard-margin SVM and minimum-ℓ2-norm interpolator coincide for classification tasks?
• RQ4 How do feature spectra and anisotropy influence the generalization of interpolating predictors in classification and regression?
• RQ5 Can a regime exist where minimum-ℓ2-norm interpolators generalize well for classification but poorly for regression?

Key findings

• In sufficiently overparameterized settings, every training point becomes a support vector, making hinge, logistic, and square losses yield identical interpolants (minimum-ℓ2-norm on labels).
• There exist regimes where test 0-1 loss generalizes well for classification while square-loss generalization for regression is poor (approaching null risk).
• The hard-margin SVM and the minimum-ℓ2-norm interpolator are equivalent with high probability under strong overparameterization (Theorem 11).
• Interpolation-based predictors can generalize in classification even when trained with non-hinge losses, challenging the notion that margin-based explanations suffice for generalization in overparameterized models.
• A Fourier/signal-processing perspective connects recent regression analyses to classification, providing non-asymptotic bounds for classification interpolators.
• The work demonstrates different roles for training-phase losses and testing-phase losses in overparameterized regimes.