[Paper Review] Learning from Noisy Labels with Deep Neural Networks: A Survey

Hwanjun Song, Minseok Kim+3 more|arXiv (Cornell University)|Jul 16, 2020

Machine Learning and Data ClassificationComputer Science91 citations

TL;DR

This survey reviews robust training methods for deep neural networks learning with noisy labels, categorizing 62 methods into five groups and analyzing evaluation practices and noise settings.

ABSTRACT

Deep learning has achieved remarkable success in numerous domains with help from large amounts of big data. However, the quality of data labels is a concern because of the lack of high-quality labels in many real-world scenarios. As noisy labels severely degrade the generalization performance of deep neural networks, learning from noisy labels (robust training) is becoming an important task in modern deep learning applications. In this survey, we first describe the problem of learning with label noise from a supervised learning perspective. Next, we provide a comprehensive review of 62 state-of-the-art robust training methods, all of which are categorized into five groups according to their methodological difference, followed by a systematic comparison of six properties used to evaluate their superiority. Subsequently, we perform an in-depth analysis of noise rate estimation and summarize the typically used evaluation methodology, including public noisy datasets and evaluation metrics. Finally, we present several promising research directions that can serve as a guideline for future studies. All the contents will be available at https://github.com/songhwanjun/Awesome-Noisy-Labels.

Motivation & Objective

• Motivate the problem of learning from noisy labels in deep learning and its impact on generalization.
• Provide a comprehensive taxonomy of robust training methods for DNNs dealing with label noise.
• Summarize theoretical foundations and practical algorithms across architecture, regularization, loss functions, loss adjustment, and sample selection.
• Evaluate how these methods are tested, including noise models and public datasets, and discuss measurement of robustness.
• Suggest promising research directions to guide future work in robust deep learning under label noise.

Proposed method

• Categorize robust training approaches into five groups: robust architecture, robust regularization, robust loss function, loss adjustment, and sample selection.
• Describe architectural adaptations such as a noise adaptation layer to model label transition probabilities and dedicated architectures for complex noise.
• Explain explicit and implicit regularization techniques including bilevel learning, pre-training, mixup, and data augmentation.
• Summarize robust loss functions such as MAE, GCE, bi-tempered loss, symmetric cross entropy, curriculum loss, and active/passive loss.
• Detail loss adjustment methods that modify the loss using estimated noise transition matrices, including backward/forward correction and gold loss correction.
• Discuss sample selection strategies and meta-learning approaches that identify clean labels and adjust learning processes accordingly.
• Reference theoretical results showing conditions under which certain losses are robust to symmetric or asymmetric noise and how these relate to deep networks.

Experimental results

Research questions

• RQ1 What are the main categories of robust training techniques for deep neural networks under label noise?
• RQ2 How do noise models (symmetric, asymmetric, instance-dependent) affect the design and effectiveness of robust methods?
• RQ3 What are the theoretical guarantees or conditions for loss functions to be robust to label noise in deep learning?
• RQ4 How are robust methods evaluated on public datasets and with different noise regimes, and what metrics are used?
• RQ5 What are promising directions and gaps for future research in learning from noisy labels with DNNs?

Key findings

• The survey reviews 62 state-of-the-art robust training methods categorized into five groups.
• Robust architecture approaches include noise adaptation layers and dedicated architectures to model label transition probabilities.
• Regularization techniques (explicit and implicit) enhance robustness, including bilevel learning, pre-training, adversarial training, and mixup.
• Robust loss functions (e.g., MAE, GCE, SCE) and loss adjustment methods (forward/backward correction, gold loss correction) are analyzed for noise tolerance.
• Loss adjustment and sample selection, often combined with meta-learning, are important for mitigating the impact of noisy labels.
• The paper discusses noise rate estimation, evaluation methodology, public noisy datasets, and evaluation metrics, highlighting directions for future research.