[Paper Review] Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition

Ben Adlam, Jeffrey Pennington|arXiv (Cornell University)|Nov 4, 2020

Machine Learning in Materials ScienceMaterials Science22 citations

TL;DR

This paper introduces a symmetric, fine-grained bias-variance decomposition that isolates variance contributions from data sampling, model initialization, and label noise in overparameterized models. It reveals that double descent arises not from overall variance but from interaction terms—particularly $V_{PX}$ and $V_{PXoldsymbol{ ho}}$—that diverge at the interpolation threshold, explaining the test error spike and showing it can be eliminated via bagging or ensembling.

ABSTRACT

Classical learning theory suggests that the optimal generalization performance of a machine learning model should occur at an intermediate model complexity, with simpler models exhibiting high bias and more complex models exhibiting high variance of the predictive function. However, such a simple trade-off does not adequately describe deep learning models that simultaneously attain low bias and variance in the heavily overparameterized regime. A primary obstacle in explaining this behavior is that deep learning algorithms typically involve multiple sources of randomness whose individual contributions are not visible in the total variance. To enable fine-grained analysis, we describe an interpretable, symmetric decomposition of the variance into terms associated with the randomness from sampling, initialization, and the labels. Moreover, we compute the high-dimensional asymptotic behavior of this decomposition for random feature kernel regression, and analyze the strikingly rich phenomenology that arises. We find that the bias decreases monotonically with the network width, but the variance terms exhibit non-monotonic behavior and can diverge at the interpolation boundary, even in the absence of label noise. The divergence is caused by the \emph{interaction} between sampling and initialization and can therefore be eliminated by marginalizing over samples (i.e. bagging) \emph{or} over the initial parameters (i.e. ensemble learning).

Motivation & Objective

• To resolve the limitations of classical bias-variance decomposition in explaining double descent in overparameterized models.
• To identify the root cause of the test error spike near the interpolation threshold, which classical theory fails to explain.
• To develop a symmetric, interpretable variance decomposition that accounts for multiple sources of randomness: sampling, initialization, and label noise.
• To demonstrate that the divergence in test error is due to interaction terms between sampling and initialization, not label noise or individual sources.
• To show that ensemble methods like bagging or model averaging can eliminate the problematic variance components.

Proposed method

• Introduces a symmetric, multivariate bias-variance decomposition that avoids arbitrary conditioning order by using ANOVA-style orthogonal decomposition.
• Applies the decomposition to random feature kernel regression, enabling high-dimensional asymptotic analysis.
• Derives explicit expressions for bias and variance components: $B$, $V_P$, $V_X$, $V_{Xoldsymbol{ ho}}$, $V_{PX}$, and $V_{PXoldsymbol{ ho}}$.
• Performs asymptotic analysis in the limit of large width and sample size, identifying the behavior of each variance term.
• Uses the decomposition to isolate divergent terms at the interpolation threshold ($\phi = \psi$) and analyze their dependence on model parameters.
• Compares the proposed symmetric decomposition with prior sequential decompositions (e.g., d'Ascoli et al.), highlighting interpretational flaws in attributing divergences to single sources.

Experimental results

Research questions

• RQ1 Why does test error spike at the interpolation threshold in overparameterized models, despite low bias?
• RQ2 What specific sources of randomness contribute to the non-monotonic variance behavior in double descent?
• RQ3 Can a symmetric, multivariate variance decomposition clarify the origin of the error spike better than sequential decompositions?
• RQ4 Why do standard ensemble methods like bagging or ensembling reduce the test error spike near interpolation?
• RQ5 How do interaction terms between sampling and initialization give rise to divergent variance components?

Key findings

• The bias decreases monotonically with increasing model width, consistent with classical theory.
• The variance components $V_{PX}$ and $V_{PXoldsymbol{ ho}}$ diverge at the interpolation threshold ($\phi = \psi$), even in the absence of label noise.
• The divergence is caused by the interaction between sampling and initialization, not by label noise or individual sources.
• The divergent terms $V_{PX}$ and $V_{PXoldsymbol{ ho}}$ can be eliminated by marginalizing over training samples (bagging) or over initial parameters (ensemble learning).
• The symmetric decomposition uniquely identifies the interaction terms as the root cause, unlike prior sequential decompositions that misattribute divergences.
• The total variance and bias remain finite, but the spike in test error arises from unaccounted interaction effects in the classical decomposition.