[Paper Review] Reconciling modern machine learning and the bias-variance trade-off

Mikhail Belkin, Daniel Hsu+2 more|arXiv (Cornell University)|Dec 28, 2018

Machine Learning and Data Classification93 citations

TL;DR

This paper resolves the contradiction between classical machine learning theory and modern practice by introducing a 'double descent' curve that extends the traditional U-shaped bias-variance trade-off. It shows that increasing model capacity beyond interpolation—where models fit training data exactly—can still improve generalization, explaining why overparameterized models like deep neural networks generalize well in practice.

ABSTRACT

Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in the modern machine learning practice. The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns. However, in the modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners. In this paper, we reconcile the classical understanding and the modern practice within a unified performance curve. This curve subsumes the textbook U-shaped bias-variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. We provide evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets, and we posit a mechanism for its emergence. This connection between the performance and the structure of machine learning models delineates the limits of classical analyses, and has implications for both the theory and practice of machine learning.

Motivation & Objective

• To resolve the apparent contradiction between classical bias-variance theory and modern machine learning practice, where overparameterized models generalize well despite interpolating training data.
• To demonstrate that the classical U-shaped bias-variance curve is insufficient to explain the performance of modern models, especially deep neural networks.
• To establish the existence and ubiquity of a 'double descent' generalization error curve across diverse models and datasets.
• To provide a theoretical mechanism explaining why generalization error decreases after the interpolation threshold, challenging classical assumptions about overfitting.
• To redefine the theoretical foundations of machine learning by extending classical analyses to account for overparameterized models.

Proposed method

• Proposes a unified performance curve that extends the classical U-shaped bias-variance trade-off into a double descent curve by analyzing generalization error as a function of model capacity.
• Analyzes generalization error across increasing model capacity, including the interpolation regime where models perfectly fit training data.
• Employs empirical analysis on a wide range of models (e.g., linear models, neural networks) and datasets to demonstrate the double descent phenomenon.
• Introduces a theoretical framework linking model capacity, data structure, and generalization error, showing that double descent emerges due to the interplay between model complexity and data geometry.
• Uses statistical learning theory to formalize how overparameterized models can generalize well when the data lies in low-dimensional manifolds.
• Demonstrates that the double descent curve arises naturally when model capacity exceeds the number of training samples, particularly in high-dimensional settings.

Experimental results

Research questions

• RQ1 Why do overparameterized models such as deep neural networks generalize well despite interpolating training data, contradicting classical bias-variance theory?
• RQ2 Does the classical U-shaped bias-variance curve adequately describe the generalization behavior of modern machine learning models?
• RQ3 Under what conditions does the double descent phenomenon emerge in machine learning models across different architectures and datasets?
• RQ4 What structural or mathematical properties of models and data give rise to the double descent behavior?
• RQ5 How can the classical understanding of generalization be extended to account for the success of modern overparameterized models?

Key findings

• The double descent curve is empirically observed across a wide range of models, including linear models, kernel methods, and deep neural networks, on various datasets.
• Generalization error decreases not only as model capacity increases up to the interpolation threshold but continues to decrease beyond it, contradicting classical theory.
• The double descent phenomenon is robust and appears when model capacity exceeds the number of training samples, particularly in high-dimensional settings.
• The emergence of double descent is linked to the geometric structure of data and the inductive biases of models, especially when data lies on low-dimensional manifolds.
• Overparameterized models can generalize well because they implicitly minimize a regularized risk, favoring solutions with low complexity in the data's intrinsic structure.
• The classical bias-variance trade-off is subsumed by the double descent curve, which provides a more complete and accurate description of model generalization performance.