[Paper Review] Reconciling modern machine learning practice and the bias-variance trade-off

Mikhail Belkin, Daniel Hsu+2 more|arXiv (Cornell University)|Dec 28, 2018

Machine Learning and Data ClassificationComputer Science38 references83 citations

TL;DR

The paper introduces the double descent risk curve, explaining how increasing model capacity beyond interpolation can reduce test risk, reconciling classical bias-variance theory with modern interpolating predictors across neural networks, random features, and ensemble methods.

ABSTRACT

Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in the modern machine learning practice. The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns. However, in the modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners. In this paper, we reconcile the classical understanding and the modern practice within a unified performance curve. This "double descent" curve subsumes the textbook U-shaped bias-variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. We provide evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets, and we posit a mechanism for its emergence. This connection between the performance and the structure of machine learning models delineates the limits of classical analyses, and has implications for both the theory and practice of machine learning.

Motivation & Objective

• Motivate the apparent mismatch between the bias-variance trade-off and modern interpolating models.
• Propose and describe the double descent risk curve as a unifying framework for model capacity and generalization.
• Demonstrate, via experiments, the ubiquity of double descent across neural networks, random features, and ensemble methods.
• Offer insights into the inductive biases and optimization dynamics that drive this behavior.

Proposed method

• Define the classical bias-variance framework and the interpolation threshold.
• Introduce Random Fourier Features as a controllable model class to study capacity.
• Train models with ERM on squared loss and compare across varying capacity (N) including N<n and N≥n.
• Show that kernel/ minimum-norm interpolants (H_infty) often generalize better beyond interpolation than finite-N classes.
• Extend observations to neural networks and ensemble methods (AdaBoost, Random Forests) showing similar double descent curves.
• Provide intuition that larger capacity enables finding simpler, smaller-norm interpolants that generalize better.

Experimental results

Research questions

• RQ1 Does a double descent risk curve emerge when increasing model capacity beyond the interpolation threshold?
• RQ2 Is double descent ubiquitous across model classes such as neural networks, random features, and tree-based ensembles?
• RQ3 What inductive biases or norms (e.g., minimum-norm solutions) underlie the better generalization seen beyond interpolation?

Key findings

• Double descent generalization curves: increasing capacity past interpolation first worsens and then improves test risk.
• Minimum-norm interpolants (or smoother averaged/interpolated solutions) tend to yield better generalization beyond interpolation, explaining the second descent.
• Random Fourier Features experiments show a peak at the interpolation threshold (N=n) with improved test performance for N>n.
• Neural networks, including two-layer nets and multi-layer architectures, exhibit a qualitatively similar double descent pattern, with optimization dynamics affecting observability.
• Ensemble methods like AdaBoost and Random Forests also display double descent when using highly interpolating trees, with averaging contributing to smoother generalization.
• The kernel limit (H_infty) provides a benchmark that often outperforms finite-N random feature models, highlighting a consistency with minimum-norm interpolation across regimes.