[Paper Review] A Modern Take on the Bias-Variance Tradeoff in Neural Networks

Brady Neal, Sarthak Mittal+5 more|arXiv (Cornell University)|Oct 19, 2018

Neural Networks and Applications55 references74 citations

TL;DR

The authors measure bias and variance in modern neural networks and show that both can decrease as width increases, challenging the classic bias-variance tradeoff. They also decompose variance into sampling and optimization components and provide supporting theory.

ABSTRACT

The bias-variance tradeoff tells us that as model complexity increases, bias falls and variances increases, leading to a U-shaped test error curve. However, recent empirical results with over-parameterized neural networks are marked by a striking absence of the classic U-shaped test error curve: test error keeps decreasing in wider networks. This suggests that there might not be a bias-variance tradeoff in neural networks with respect to network width, unlike was originally claimed by, e.g., Geman et al. (1992). Motivated by the shaky evidence used to support this claim in neural networks, we measure bias and variance in the modern setting. We find that both bias and variance can decrease as the number of parameters grows. To better understand this, we introduce a new decomposition of the variance to disentangle the effects of optimization and data sampling. We also provide theoretical analysis in a simplified setting that is consistent with our empirical findings.

Motivation & Objective

• Motivate the traditional bias-variance lens and its applicability to modern neural networks.
• Empirically measure prediction bias and variance as network width increases on diverse datasets.
• Decompose total variance into sampling and optimization components to disentangle their effects.
• Provide theoretical analysis that aligns with the empirical observations in simplified settings.

Proposed method

• Define a bias-variance decomposition for squared-loss predictions and adapt it to classification contexts.
• Introduce a variance decomposition using the law of total variance to separate sampling vs optimization variance.
• Conduct extensive experiments across MNIST, CIFAR-10, SVHN, small MNIST, and a sinusoid regression task with varying widths.
• Use bootstrap replicates to estimate bias and variance terms and 10/10 seeds for outer/inner expectations, reporting 99% CIs.
• Demonstrate that variance due to optimization decreases with width while variance due to sampling plateaus.
• Provide theoretical insights in over-parameterized linear models and related settings to support the empirical findings.

Experimental results

Research questions

• RQ1 Does prediction bias decrease as neural network width increases in practical settings?
• RQ2 How does prediction variance behave with increasing width: does it increase, decrease, or plateau?
• RQ3 Can we disentangle variance into optimization-driven and sampling-driven components, and how do these behave with width?
• RQ4 Do linear or simplified models offer insights that explain the observed bias-variance behavior in wide networks?

Key findings

• Both bias and variance can decrease as network width increases across multiple datasets.
• Variance due to optimization decreases with width and can become negligible in the over-parameterized regime.
• Variance due to sampling increases slowly and tends to plateau once sufficiently over-parameterized.
• The law of total variance enables a decomposition into an optimization-driven term and a sampling-driven term, clarifying their distinct roles.
• Empirical results are complemented by theoretical analysis in simplified settings that are consistent with the observations.