[Paper Review] Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels

Lu Jiang, Di Huang+2 more|arXiv (Cornell University)|Nov 21, 2019

Machine Learning and Data ClassificationComputer Science123 citations

TL;DR

The paper introduces a first benchmark for controlled real-world (web) label noise, proposes MentorMix to handle both synthetic and real noisy labels, and presents large-scale findings on how DNNs learn with noisy labels across noise types, levels, architectures, and training settings.

ABSTRACT

Performing controlled experiments on noisy data is essential in understanding deep learning across noise levels. Due to the lack of suitable datasets, previous research has only examined deep learning on controlled synthetic label noise, and real-world label noise has never been studied in a controlled setting. This paper makes three contributions. First, we establish the first benchmark of controlled real-world label noise from the web. This new benchmark enables us to study the web label noise in a controlled setting for the first time. The second contribution is a simple but effective method to overcome both synthetic and real noisy labels. We show that our method achieves the best result on our dataset as well as on two public benchmarks (CIFAR and WebVision). Third, we conduct the largest study by far into understanding deep neural networks trained on noisy labels across different noise levels, noise types, network architectures, and training settings. The data and code are released at the following link: http://www.lujiang.info/cnlw.html

Motivation & Objective

• Develop a controlled benchmark for web (real-world) label noise by annotating web images across multiple noise levels.
• Propose and validate a robust learning method (MentorMix) that handles both synthetic and real noisy labels without clean labels.
• Empirically compare MentorMix to state-of-the-art baselines on public synthetic and real-world noisy-label benchmarks.
• Analyze how DNNs behave when trained on noisy labels across noise types, levels, architectures, and training settings to refine understanding in this area.

Proposed method

• Introduce MentorMix, a robust loss based on empirical vicinal risk minimization combined with curriculum learning.
• Compute an optimal latent weight for each training example via MentorNet, guiding a weighted mixup of samples.
• Use a practical importance-sampling scheme to select mixed-up pairs biased toward lower-loss examples.
• Leverage a moving-average percentile of per-sample loss to modulate example weights and stabilize training.
• Demonstrate that MentorMix outperforms baselines on controlled red (web) and blue (synthetic) noise across datasets and training settings.

Experimental results

Research questions

• RQ1 Can controlled web-label noise be constructed and used to study DNNs across multiple noise levels in a controlled setting?
• RQ2 Does the MentorMix approach provide robust performance against both synthetic and real-world noisy labels without access to clean labels?
• RQ3 How does MentorMix perform relative to state-of-the-art methods on public benchmarks with synthetic and real-world noisy labels?
• RQ4 What are the broader patterns of DNN behavior when trained with noisy labels across different noise types, levels, architectures, and training regimes?

Key findings

• Established the first benchmark of controlled web label noise (red noise) with manual annotations across 10 noise levels on Mini-ImageNet and Stanford Cars.
• MentorMix consistently outperforms baselines on both synthetic and web noisy labels and achieves state-of-the-art results on CIFAR and WebVision benchmarks.
• On WebVision 1.0, MentorMix improves top-1 accuracy by about 3% over prior methods without extra clean labels.
• MentorMix yields notable improvements over vanilla training and existing robust methods, including large gains in noisy settings across multiple architectures.
• The study confirms prior findings about neural networks with synthetic noise and offers new observations that challenge common intuitions about learning with noisy labels.