[Paper Review] Deep Learning is Robust to Massive Label Noise

David Rolnick, Andreas Veit+2 more|arXiv (Cornell University)|May 30, 2017

Machine Learning and Data ClassificationComputer Science27 references408 citations

TL;DR

The paper shows standard deep networks can generalize well even when training labels are massively corrupted, given sufficiently large amounts of data and appropriate training settings.

ABSTRACT

Deep neural networks trained on large supervised datasets have led to impressive results in image classification and other tasks. However, well-annotated datasets can be time-consuming and expensive to collect, lending increased interest to larger but noisy datasets that are more easily obtained. In this paper, we show that deep neural networks are capable of generalizing from training data for which true labels are massively outnumbered by incorrect labels. We demonstrate remarkably high test performance after training on corrupted data from MNIST, CIFAR, and ImageNet. For example, on MNIST we obtain test accuracy above 90 percent even after each clean training example has been diluted with 100 randomly-labeled examples. Such behavior holds across multiple patterns of label noise, even when erroneous labels are biased towards confusing classes. We show that training in this regime requires a significant but manageable increase in dataset size that is related to the factor by which correct labels have been diluted. Finally, we provide an analysis of our results that shows how increasing noise decreases the effective batch size.

Motivation & Objective

• Motivate the study of deep learning under extreme label noise to assess practical data acquisition with noisy annotations.
• Quantify how test performance degrades (or remains) as label noise increases across multiple datasets and architectures.
• Understand how dataset size, batch size, and learning rate interact to sustain learning under noise.
• Characterize the sources and structures of label noise and their impact on learning dynamics.
• Provide insights for designing training pipelines when clean labels are scarce or expensive.

Proposed method

• Systematically add α noisy labels per clean label to create highly noisy training sets across MNIST, CIFAR-10, and ImageNet.
• Evaluate various architectures (MLP, ConvNet, ResNet) to compare noise tolerance.
• Model three noise patterns: uniform label noise, structured noise, and out-of-vocabulary/noise from other sources.
• Analyze the effect of batch size and learning rate on learning under noise using empirical experiments and a theoretical noisy loss formulation (Hα).
• Investigate the influence of different noise sources (same dataset, different dataset, white noise) on robustness.

Experimental results

Research questions

• RQ1 How well can deep networks generalize when label noise vastly outnumbers correct labels?
• RQ2 What dataset size, batch size, and learning rate settings are needed to achieve high accuracy under massive label noise?
• RQ3 How do different noise structures (uniform, structured) affect learning with noisy labels?
• RQ4 What is the impact of the noise source (in-domain vs out-of-domain vs random) on robustness?
• RQ5 Can a theoretical noisy-loss perspective explain observed robustness under high noise?

Key findings

• Deep networks maintain high test performance even when correct labels are outnumbered by noisy labels by factors up to 100 on MNIST (over 90% accuracy).
• On CIFAR-10, networks achieve over 85% accuracy with 10 noisy labels per clean label; on ImageNet, top-5 accuracy remains above 70% with 5 noisy labels per clean label.
• Larger networks (ConvNets, ResNets) tend to be more robust to label noise than smaller architectures.
• A sufficiently large training set compensates for higher noise, with the required amount of clean data increasing roughly linearly with noise level (slightly super-linear).
• Increased batch size mitigates the effective loss of signal from noisy labels, and learning rate should be scaled down as noise increases.