[论文解读] Label Noise Types and Their Effects on Deep Learning

Görkem Algan, İlkay Ulusoy|arXiv (Cornell University)|Mar 23, 2020

Machine Learning and Data Classification参考文献 40被引用 40

一句话总结

本文分析了统一、类依赖、局部集中、以及特征相关的标签噪声如何影响深度学习，并提出一个使用蒸馏来创建稀疏特征空间表示的通用特征相关噪声生成器。

ABSTRACT

The recent success of deep learning is mostly due to the availability of big datasets with clean annotations. However, gathering a cleanly annotated dataset is not always feasible due to practical challenges. As a result, label noise is a common problem in datasets, and numerous methods to train deep neural networks in the presence of noisy labels are proposed in the literature. These methods commonly use benchmark datasets with synthetic label noise on the training set. However, there are multiple types of label noise, and each of them has its own characteristic impact on learning. Since each work generates a different kind of label noise, it is problematic to test and compare those algorithms in the literature fairly. In this work, we provide a detailed analysis of the effects of different kinds of label noise on learning. Moreover, we propose a generic framework to generate feature-dependent label noise, which we show to be the most challenging case for learning. Our proposed method aims to emphasize similarities among data instances by sparsely distributing them in the feature domain. By this approach, samples that are more likely to be mislabeled are detected from their softmax probabilities, and their labels are flipped to the corresponding class. The proposed method can be applied to any clean dataset to synthesize feature-dependent noisy labels. For the ease of other researchers to test their algorithms with noisy labels, we share corrupted labels for the most commonly used benchmark datasets. Our code and generated noisy synthetic labels are available online.

研究动机与目标

• 激励并对不同标签噪声类型及其对学习的不同影响进行分类。
• 提供一个通用的、基于蒸馏的框架来生成特征相关的标签噪声。
• 评估每种噪声类型如何影响在基准数据集上的训练、验证和测试性能。
• 分享常用数据集的预生成嘈声标签，以帮助公平测试抗噪声方法。

提出的方法

• 对三大主要噪声类别进行调研和形式化：统一、类依赖和特征依赖。
• 提出一种受知识蒸馏启发的特征相关噪声生成方法，用于创建稀疏的特征空间表示。
• 使用教师-学生蒸馏框架根据 softmax 概率识别并翻转不确定样本的标签。
• 通过类特定特征表示的方差来量化稀疏性，并与标准训练进行比较。
• 为 MNIST、Fashion-MNIST、CIFAR-10 和 CIFAR-100 数据集提供预生成的嘈声标签。

实验结果

研究问题

• RQ1 不同标签噪声类型对学习动态和泛化的特征性影响是什么？
• RQ2 一个通用框架是否能够生成对标准训练具有挑战性的现实感的特征相关标签噪声？
• RQ3 特征相关噪声在训练、验证和测试性能上与统一噪声和类依赖噪声相比有何差异？
• RQ4 现有的抗噪声算法在不同噪声类型上表现是否不同，哪些类型最具挑战性？

主要发现

• 统一噪声和类依赖噪声以可预测的方式降低测试性能，但特征相关噪声尽管训练/验证性能相当，仍可能对测试准确率造成更大损害。
• 特征相关噪声往往保持训练和验证准确性，而显著降低测试准确性，使对有噪声标签的过拟合评估更为复杂。
• 局部集中噪声在更复杂的数据集（如 CIFAR-100）上可能特别有害，减慢正确学习决策边界。
• 增加数据集规模可以缓解某些噪声类型的影响，但特征相关噪声仍然非常具有挑战性。
• 抗噪声方法（如 Co-teaching）在类型上表现出差异性，没有一种方法在所有噪声类型上都出色。