[Paper Review] Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations

Jiaheng Wei, Zhaowei Zhu+4 more|arXiv (Cornell University)|Oct 22, 2021

Machine Learning and Data Classification45 references62 citations

TL;DR

Introduces CIFAR-10N and CIFAR-100N benchmarks with real-world human-annotated noisy labels, showing instance-dependent human noise differs from synthetic class-dependent noise and affecting robustness evaluations.

ABSTRACT

Existing research on learning with noisy labels mainly focuses on synthetic label noise. Synthetic noise, though has clean structures which greatly enabled statistical analyses, often fails to model real-world noise patterns. The recent literature has observed several efforts to offer real-world noisy datasets, yet the existing efforts suffer from two caveats: (1) The lack of ground-truth verification makes it hard to theoretically study the property and treatment of real-world label noise; (2) These efforts are often of large scales, which may result in unfair comparisons of robust methods within reasonable and accessible computation power. To better understand real-world label noise, it is crucial to build controllable and moderate-sized real-world noisy datasets with both ground-truth and noisy labels. This work presents two new benchmark datasets CIFAR-10N, CIFAR-100N, equipping the training datasets of CIFAR-10, CIFAR-100 with human-annotated real-world noisy labels we collected from Amazon Mechanical Turk. We quantitatively and qualitatively show that real-world noisy labels follow an instance-dependent pattern rather than the classically assumed and adopted ones (e.g., class-dependent label noise). We then initiate an effort to benchmarking a subset of the existing solutions using CIFAR-10N and CIFAR-100N. We further proceed to study the memorization of correct and wrong predictions, which further illustrates the difference between human noise and class-dependent synthetic noise. We show indeed the real-world noise patterns impose new and outstanding challenges as compared to synthetic label noise. These observations require us to rethink the treatment of noisy labels, and we hope the availability of these two datasets would facilitate the development and evaluation of future learning with noisy label solutions. Datasets and leaderboards are available at http://noisylabels.com.

Motivation & Objective

• Motivate the study of real-world label noise beyond synthetic models and provide accessible benchmarks with ground-truth and noise labels.
• Characterize the distribution and patterns of human-annotated noise on CIFAR-10 and CIFAR-100.
• Benchmark a broad set of robust learning methods on real human noise and compare to synthetic noise.
• Investigate memorization behaviors under real-world noisy supervision to understand learning dynamics.

Proposed method

• Create CIFAR-10N and CIFAR-100N by collecting three human annotations per image via Amazon Mechanical Turk for CIFAR-10 and one per image for CIFAR-100, with ground-truth CIFAR labels kept for evaluation.
• Analyze noise patterns to show instance-dependent, imbalanced, and feature-correlated label flips across datasets.
• Compare human-annotated noise to synthetic class-dependent noise using noise transition matrices and hypothesis testing to assess feature-dependence.
• Evaluate a broad suite of robust learning methods (loss correction, re-weighting, regularization, semi-supervised–style sampling) on CIFAR-10N and CIFAR-100N.
• Examine memorization behavior under real-world noise to reveal differences from synthetic-noise scenarios.

Experimental results

Research questions

• RQ1 Do real-world human annotations exhibit instance-dependent noise patterns that diverge from class-dependent synthetic noise?
• RQ2 How do CIFAR-10N and CIFAR-100N noise distributions differ in terms of imbalances and label flips among similar classes?
• RQ3 How do popular robust learning methods perform on real-world noisy labels versus synthetic noise?
• RQ4 What memorization dynamics arise when training with human-generated label noise compared to synthetic noise?

Key findings

• Real-world human noise is instance-dependent and exhibits imbalanced, feature-correlated label flips not well captured by synthetic class-dependent noise.
• Humans tend to mislabel among visually similar classes, and multiple clean labels can co-exist for CIFAR-100 images, creating novel noise patterns.
• Across methods, real-world noise presents a larger learning challenge than synthetic noise, with notable performance gaps and memorization differences.
• Some methods (e.g., ELR+, Divide-Mix) show robust performance on synthetic noise but exhibit different behavior under human noise, and in some settings human noise can even facilitate slightly better performance for certain approaches.
• Memorization studies reveal neural networks tend to memorize wrong labels more readily under human noise than under synthetic noise, indicating different learning dynamics.