[论文解读] Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations

Jiaheng Wei, Zhaowei Zhu+4 more|arXiv (Cornell University)|Oct 22, 2021

Machine Learning and Data Classification参考文献 45被引用 62

一句话总结

简述：引入带有真实世界人工标注噪声标签的 CIFAR-10N 与 CIFAR-100N 基准数据集，显示实例相关的人类噪声与合成的类相关噪声存在差异，影响鲁棒性评估。

ABSTRACT

Existing research on learning with noisy labels mainly focuses on synthetic label noise. Synthetic noise, though has clean structures which greatly enabled statistical analyses, often fails to model real-world noise patterns. The recent literature has observed several efforts to offer real-world noisy datasets, yet the existing efforts suffer from two caveats: (1) The lack of ground-truth verification makes it hard to theoretically study the property and treatment of real-world label noise; (2) These efforts are often of large scales, which may result in unfair comparisons of robust methods within reasonable and accessible computation power. To better understand real-world label noise, it is crucial to build controllable and moderate-sized real-world noisy datasets with both ground-truth and noisy labels. This work presents two new benchmark datasets CIFAR-10N, CIFAR-100N, equipping the training datasets of CIFAR-10, CIFAR-100 with human-annotated real-world noisy labels we collected from Amazon Mechanical Turk. We quantitatively and qualitatively show that real-world noisy labels follow an instance-dependent pattern rather than the classically assumed and adopted ones (e.g., class-dependent label noise). We then initiate an effort to benchmarking a subset of the existing solutions using CIFAR-10N and CIFAR-100N. We further proceed to study the memorization of correct and wrong predictions, which further illustrates the difference between human noise and class-dependent synthetic noise. We show indeed the real-world noise patterns impose new and outstanding challenges as compared to synthetic label noise. These observations require us to rethink the treatment of noisy labels, and we hope the availability of these two datasets would facilitate the development and evaluation of future learning with noisy label solutions. Datasets and leaderboards are available at http://noisylabels.com.

研究动机与目标

• 激发对现实世界标签噪声的研究，而不仅仅是合成模型，并提供可访问的带有真实标签和噪声标签的基准数据。
• 描述 CIFAR-10 和 CIFAR-100 上人工标注噪声的分布和模式。
• 在真实的人类噪声上对广泛的鲁棒学习方法进行基准评估，并与合成噪声进行比较。
• 研究在真实世界嘈杂监督下的记忆化行为，以理解学习动态。

提出的方法

• 通过 Amazon Mechanical Turk 为 CIFAR-10 收集每张图像三次人工标注，为 CIFAR-100 收集每张图像一次人工标注，并保留 CIFAR 的真实标签用于评估。
• 分析噪声模式，展示跨数据集的实例相关、失衡以及特征相关的标签翻转。
• 使用噪声转移矩阵和假设检验，将人类标注的噪声与合成的类相关噪声进行比较以评估特征依赖性。
• 在 CIFAR-10N 和 CIFAR-100N 上评估广泛的鲁棒学习方法（损失校正、重加权、正则化、半监督风格采样）。
• 检视在真实世界噪声下的记忆行为，以揭示与合成噪声情景的差异。

实验结果

研究问题

• RQ1 现实世界的人类标注是否展现出实例相关的噪声模式，偏离类相关的合成噪声？
• RQ2 在相似类别之间的不平衡和标签翻转方面，CIFAR-10N 与 CIFAR-100N 的噪声分布有何差异？
• RQ3 受欢迎的鲁棒学习方法在现实世界带噪标签与合成噪声下的表现如何？
• RQ4 与合成噪声相比，使用人工生成的标签噪声训练时会出现哪些记忆化动力学？

主要发现

• 现实世界的人类噪声具有实例相关性，并且表现出不平衡、特征相关的标签翻转，未被合成的类相关噪声很好捕捉。
• 人们倾向于在视觉上相似的类别之间标错标签，且 CIFAR-100 的图像可能存在多个干净标签共存，产生新的噪声模式。
• 在各方法中，现实世界噪声带来比合成噪声更大的学习挑战，存在显著的性能差距和记忆化差异。
• 某些方法（例如 ELR+、Divide-Mix）在合成噪声上表现稳健，但在真实噪声下表现不同，在某些设置下人类噪声甚至可以略微提升某些方法的性能。
• 记忆化研究显示神经网络在真实世界噪声下更容易记住错误标签，程度高于合成噪声，表明学习动力学不同。