[Paper Review] Rethinking Bias-Variance Trade-off for Generalization of Neural Networks

Zitong Yang, Yaodong Yu+3 more|arXiv (Cornell University)|Feb 26, 2020

Adversarial Robustness in Machine Learning31 references81 citations

TL;DR

The paper shows that in neural networks the bias decreases with width while the variance is unimodal (increases then decreases), leading to various risk curves including monotonic, double-descent, and unimodal, with theoretical support from a two-layer linear model.

ABSTRACT

The classical bias-variance trade-off predicts that bias decreases and variance increase with model complexity, leading to a U-shaped risk curve. Recent work calls this into question for neural networks and other over-parameterized models, for which it is often observed that larger models generalize better. We provide a simple explanation for this by measuring the bias and variance of neural networks: while the bias is monotonically decreasing as in the classical theory, the variance is unimodal or bell-shaped: it increases then decreases with the width of the network. We vary the network architecture, loss function, and choice of dataset and confirm that variance unimodality occurs robustly for all models we considered. The risk curve is the sum of the bias and variance curves and displays different qualitative shapes depending on the relative scale of bias and variance, with the double descent curve observed in recent literature as a special case. We corroborate these empirical results with a theoretical analysis of two-layer linear networks with random first layer. Finally, evaluation on out-of-distribution data shows that most of the drop in accuracy comes from increased bias while variance increases by a relatively small amount. Moreover, we find that deeper models decrease bias and increase variance for both in-distribution and out-of-distribution data.

Motivation & Objective

• Explain why larger neural networks often generalize better despite classical bias-variance expectations.
• Empirically characterize how bias and variance depend on network width, architecture, loss function, and dataset.
• Provide a theoretical analysis validating unimodal variance in a simplified model.
• Examine implications for out-of-distribution performance and model depth on bias/variance.
• Relate findings to the double-descent phenomenon observed in modern ML.

Proposed method

• Define random-design bias-variance decomposition for both squared loss and cross-entropy loss using generalized Bregman divergence.
• Empirically measure bias and variance by training multiple models on different data splits and estimating variance with an unbiased estimator.
• Vary network width, architecture, loss function, and dataset to observe bias monotonicity and variance unimodality.
• Conduct a theoretical analysis of a two-layer linear network with random first layer to show monotone bias and unimodal variance in an asymptotic regime.
• Analyze the effect of depth on bias and variance and study out-of-distribution performance using CIFAR10-C.

Experimental results

Research questions

• RQ1 How does bias behave as network width increases in neural networks?
• RQ2 How does variance behave with network width across architectures and losses?
• RQ3 What shapes do the overall risk curves take given the relative scales of bias and variance?
• RQ4 Do deeper networks alter the bias-variance dynamics for in-distribution and out-of-distribution data?
• RQ5 Can a simplified two-layer linear model reproduce the observed bias-variance behavior and explain peak locations?

Key findings

• Bias decreases monotonically with width, consistent with classical theory.
• Variance is unimodal (bell-shaped): it increases then decreases with width across architectures and datasets.
• Risk curves can be monotonic, double-descent, or unimodal depending on the relative sizes of bias and variance.
• Deeper models tend to reduce bias while increasing variance, suggesting depth helps generalization mainly through bias reduction.
• On out-of-distribution data, most accuracy drops stem from increased bias, with variance contributing less.
• A two-layer linear model with random first layer exhibits the same monotonic bias and unimodal variance, supporting the observed phenomena.