[Paper Review] Label Noise Types and Their Effects on Deep Learning

Görkem Algan, İlkay Ulusoy|arXiv (Cornell University)|Mar 23, 2020

Machine Learning and Data Classification40 references40 citations

TL;DR

The paper analyzes how uniform, class-dependent, locally concentrated, and feature-dependent label noise affect deep learning, and proposes a generic feature-dependent noise generator using distillation to create sparse feature-space representations.

ABSTRACT

The recent success of deep learning is mostly due to the availability of big datasets with clean annotations. However, gathering a cleanly annotated dataset is not always feasible due to practical challenges. As a result, label noise is a common problem in datasets, and numerous methods to train deep neural networks in the presence of noisy labels are proposed in the literature. These methods commonly use benchmark datasets with synthetic label noise on the training set. However, there are multiple types of label noise, and each of them has its own characteristic impact on learning. Since each work generates a different kind of label noise, it is problematic to test and compare those algorithms in the literature fairly. In this work, we provide a detailed analysis of the effects of different kinds of label noise on learning. Moreover, we propose a generic framework to generate feature-dependent label noise, which we show to be the most challenging case for learning. Our proposed method aims to emphasize similarities among data instances by sparsely distributing them in the feature domain. By this approach, samples that are more likely to be mislabeled are detected from their softmax probabilities, and their labels are flipped to the corresponding class. The proposed method can be applied to any clean dataset to synthesize feature-dependent noisy labels. For the ease of other researchers to test their algorithms with noisy labels, we share corrupted labels for the most commonly used benchmark datasets. Our code and generated noisy synthetic labels are available online.

Motivation & Objective

• Motivate and categorize different label-noise types and their distinct impacts on learning.
• Provide a generic, distillation-based framework to generate feature-dependent label noise.
• Evaluate how each noise type affects training, validation, and test performance across benchmarks.
• Share pre-generated noisy labels for common datasets to aid fair testing of noise-robust methods.

Proposed method

• Survey and formalize three main noise categories: uniform, class-dependent, and feature-dependent.
• Propose a feature-dependent noise generation method inspired by knowledge distillation to create sparse feature-space representations.
• Use a teacher-student distillation setup to identify and flip labels for uncertain samples based on softmax probabilities.
• Quantify sparsity via variance of class-specific feature representations and compare with standard training.
• Provide pre-generated noisy labels for MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets.

Experimental results

Research questions

• RQ1 What are the characteristic effects of different label-noise types on learning dynamics and generalization?
• RQ2 Can a generic framework generate realistic feature-dependent label noise that is challenging for standard training?
• RQ3 How do feature-dependent noises compare to uniform and class-dependent noises in training, validation, and test performance?
• RQ4 Do existing noise-robust algorithms perform variably across different noise types, and which types are most problematic?

Key findings

• Uniform and class-dependent noises degrade test performance in predictable ways, but feature-dependent noise can be more harmful to test accuracy despite comparable training/validation performance.
• Feature-dependent noise tends to preserve training and validation accuracy while significantly lowering test accuracy, complicating evaluation of overfitting to noisy labels.
• Locally concentrated noise can be particularly detrimental on more complex datasets like CIFAR-100, slowing correct learning of decision boundaries.
• Increasing dataset size mitigates the impact of some noise types, but feature-dependent noise remains highly challenging.
• Noise-robust methods (e.g., Co-teaching) show type-dependent effectiveness, with no single method excelling across all noise types.