[Paper Review] Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise

Dan Hendrycks, Mantas Mazeika+2 more|arXiv (Cornell University)|Feb 14, 2018

Adversarial Robustness in Machine Learning312 citations

TL;DR

The paper introduces Gold Loss Correction (GLC), a data-efficient method that uses a small trusted set to estimate a label-noise corruption matrix and corrects loss accordingly to robustly train deep networks under severe label noise.

ABSTRACT

The growing importance of massive datasets used for deep learning makes robustness to label noise a critical property for classifiers to have. Sources of label noise include automatic labeling, non-expert labeling, and label corruption by data poisoning adversaries. Numerous previous works assume that no source of labels can be trusted. We relax this assumption and assume that a small subset of the training data is trusted. This enables substantial label corruption robustness performance gains. In addition, particularly severe label noise can be combated by using a set of trusted data with clean labels. We utilize trusted data by proposing a loss correction technique that utilizes trusted examples in a data-efficient manner to mitigate the effects of label noise on deep neural network classifiers. Across vision and natural language processing tasks, we experiment with various label noises at several strengths, and show that our method significantly outperforms existing methods.

Motivation & Objective

• Motivate robust learning when training labels are heavily corrupted due to automatic labeling, non-expert labeling, or data poisoning.
• Propose a loss-correction framework that leverages a small trusted dataset to mitigate label noise.
• Demonstrate that GLC yields significant robustness gains across vision and NLP tasks compared to existing methods.

Proposed method

• Define a corruption matrix C with C_{ij}=P(tilde{y}=j | y=i) representing label noise.
• Estimate C from untrusted data by training a noisy-label classifier p(tilde{y}|x) and averaging its predictions over trusted examples for each true class i.
• Train a corrected classifier by applying C^T to the model outputs and optimizing on noisy labels, while optionally keeping trusted examples uncorrected.
• Propose Gold Loss Correction (GLC) which combines estimating C with a corrected loss to recover P(y|x).
• Compare GLC to Forward, Distillation, and Confusion-Matrix baselines, showing improved area-under-the-error-curve across datasets and noise types.
• Explore data efficiency by testing on various trusted fractions and using both synthetic and weak-label noise settings.

Experimental results

Research questions

• RQ1 How can a small set of trusted labels be leveraged to robustly learn under severe label noise?
• RQ2 Does estimating a label-noise matrix from trusted data improve recovery of the true label distribution compared to existing loss-correction methods?
• RQ3 How data-efficient is the proposed approach across vision and NLP tasks under different corruption types and strengths?
• RQ4 What is the impact of using trusted data on forward-loss corrections and distillation baselines?
• RQ5 Can GLC handle instance-dependent noise and still outperform state-of-the-art methods with limited gold labels?

Key findings

• GLC consistently achieves better robustness (lower error) than baselines across MNIST, CIFAR (10/100), IMDB, SST, Twitter POS, and other NLP tasks under uniform, flip, and hierarchical corruption.
• Estimating the corruption matrix via trusted data provides a more accurate C than confusion-matrix or forward methods, yielding stronger performance especially at moderate-to-severe noise levels.
• GLC remains data-efficient, achieving notable gains with very small trusted fractions (e.g., 1-5%), and outperforming distillation in several settings when trusted data is scarce.
• In clothing1M, GLC outperforms distillation, especially with fewer trusted examples, demonstrating favorable data efficiency in real-world noisy-labeled data scenarios.
• Weak-classifier label experiments show GLC often achieves the best error-rate reduction even when noisy labels originate from a weak model, highlighting the method’s robustness to different noise sources.
• Overall, GLC provides a state-of-the-art, data-efficient approach to learning under label noise and can leverage a small gold-standard set for substantial gains.