[Paper Review] Understanding deep learning requires rethinking generalization

Chiyuan Zhang, Samy Bengio+3 more|arXiv (Cornell University)|Nov 10, 2016

Domain Adaptation and Few-Shot Learning29 references1,089 citations

TL;DR

The paper shows that modern deep networks can memorize random labels/pixels, challenging traditional generalization theories, and proves finite-sample expressivity for simple networks, arguing implicit regularization via SGD is key.

ABSTRACT

Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family, or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization, and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models.

Motivation & Objective

• Question why large neural networks generalize despite high capacity.
• Test whether frameworks based on traditional complexity measures explain generalization in deep nets.
• Explore the role of explicit vs. implicit regularization in deep learning.
• Demonstrate finite-sample expressivity of neural networks for memorizing data.

Proposed method

• Perform randomization tests by training standard architectures on data with random labels.
• Replace true images with random pixels and vary noise level to assess fitting ability.
• Train with/without explicit regularizers (weight decay, dropout, data augmentation) and compare generalization.
• Theoretically construct a two-layer ReLU network with p = 2n + d parameters that can fit any labeling of a size-n sample in d dimensions.
• Analyze SGD as an implicit regularizer by connecting solutions to minimum-norm solutions in linear models.
• Discuss implications for VC-dimension, Rademacher complexity, and uniform stability.

Experimental results

Research questions

• RQ1 Can deep networks fit training data when labels are randomized, and what does this imply about effective capacity?
• RQ2 Do explicit regularizers fully explain generalization, or is implicit regularization via optimization more pivotal?
• RQ3 How does finite-sample expressivity constrain our understanding of network generalization?
• RQ4 What is the role of optimization dynamics (SGD) in selecting among many interpolating solutions?
• RQ5 How do standard complexity measures (VC, Rademacher, stability) align with observed deep learning generalization?

Key findings

• Deep networks can achieve zero training error on random labels across CIFAR-10 and ImageNet, indicating sufficient capacity to memorize data.
• Generalization error grows when labels are randomized, even though training error remains zero, challenging traditional complexity-based explanations.
• Explicit regularizers (weight decay, dropout, data augmentation) improve but are not essential for good generalization on these tasks.
• A simple two-layer ReLU network with 2n + d parameters can represent any labeling of an n-point sample in d dimensions, showing strong finite-sample expressivity.
• SGD acts as an implicit regularizer in linear settings, and minimum-norm solutions can generalize well, suggesting optimization dynamics contribute to generalization beyond explicit regularizers.