[Paper Review] Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate

Mikhail Belkin, Daniel Hsu+1 more|arXiv (Cornell University)|Jun 13, 2018

Adversarial Robustness in Machine LearningComputer Science2 references85 citations

TL;DR

The paper analyzes interpolating prediction schemes (like simplicial interpolation and wiNN) and proves consistency or near-consistency in classification and regression, derives non-asymptotic and minimax-optimal rates, and links interpolation to adversarial examples.

ABSTRACT

Many modern machine learning models are trained to achieve zero or near-zero training error in order to obtain near-optimal (but non-zero) test error. This phenomenon of strong generalization performance for "overfitted" / interpolated classifiers appears to be ubiquitous in high-dimensional data, having been observed in deep networks, kernel machines, boosting and random forests. Their performance is consistently robust even when the data contain large amounts of label noise. Very little theory is available to explain these observations. The vast majority of theoretical analyses of generalization allows for interpolation only when there is little or no label noise. This paper takes a step toward a theoretical foundation for interpolated classifiers by analyzing local interpolating schemes, including geometric simplicial interpolation algorithm and singularly weighted $k$-nearest neighbor schemes. Consistency or near-consistency is proved for these schemes in classification and regression problems. Moreover, the nearest neighbor schemes exhibit optimal rates under some standard statistical assumptions. Finally, this paper suggests a way to explain the phenomenon of adversarial examples, which are seemingly ubiquitous in modern machine learning, and also discusses some connections to kernel machines and random forests in the interpolated regime.

Motivation & Objective

• Motivate and develop a theoretical foundation for predictors that interpolate training data in classification and regression.
• Establish consistency or near-consistency for local interpolating schemes under standard smoothness and margin assumptions.
• Derive non-asymptotic risk bounds and, under margin conditions, improved rates including minimax-optimal rates for certain schemes.
• Provide insights into the role of interpolation in adversarial examples and connections to kernel methods and random forests.

Proposed method

• Introduce interpolating schemes: (i) simplicial interpolation based on multivariate triangulation, and (ii) weighted interpolated nearest neighbor (wiNN) with singular radial weights.
• Analyze mean squared error and classification risk by bounding the excess risk directly through locality properties and interpolation behavior.
• Prove that simplicial interpolation achieves near-Bayes risk in high dimensions under smoothness and margin conditions; provide corresponding results for wiNN with explicit rates.
• Derive non-asymptotic convergence rates; show minimax-optimal rates for wiNN under standard assumptions; discuss how dimensionality impacts rates (blessing of dimensionality).
• Establish conditions under which interpolation yields adversarial examples and discuss their abundance in the interpolated regime.

Experimental results

Research questions

• RQ1 Can interpolating classifiers and regressors achieve consistency or near-consistency under standard smoothness and margin assumptions?
• RQ2 What are the finite-sample and asymptotic risk bounds for interpolating schemes such as simplicial interpolation and wiNN?
• RQ3 How do interpolation methods perform in high dimensions, and can they achieve minimax-optimal or exponentially small excess risk under margin conditions?
• RQ4 What is the relationship between interpolation, adversarial examples, and connections to kernel methods and random forests?
• RQ5 How do direct (prediction based on interpolation) versus inverse interpolation schemes compare in terms of risk and consistency?

Key findings

• Simplicial interpolation yields a piecewise linear, continuous predictor whose risk approaches the Bayes risk as dimension grows, under regularity conditions.
• Under a simple polytope support and Delaunay triangulation, the regression error bound satisfies limsup_n E[(hat_eta(X)−eta(X))^2] ≤ (2/(d+2)) E[(Y−eta(X))^2].
• For classification, the plug-in classifier based on hat_eta achieves a bound implying the excess risk can be near the Bayes risk in high dimensions, with an exponential improvement under a hard margin condition (risk bound decays as exp(-c d)).
• The wiNN scheme with singular weights achieves minimax-optimal convergence rates n^{-2α/(2α+d)} in regression and n^{-α/(2α+d)} in classification under standard regularity, margin, and density conditions.
• The analysis shows that interpolation necessarily leads to adversarial examples in the presence of label noise: the set of adversarial basins is dense asymptotically, even though mass on misclassifications can be small.