[论文解读] FINE Samples for Learning with Noisy Labels

Taehyeon Kim, Jongwoo Ko+3 more|arXiv (Cornell University)|Feb 23, 2021

Machine Learning and Data Classification参考文献 51被引用 38

一句话总结

FINE 引入了一种无导数检测器，通过检查潜在表示与类别级 Gram 矩阵的第一特征向量的对齐来筛选标签噪声，从而实现鲁棒样本选择、SSL 以及在跨基准的鲁棒损失协作。

ABSTRACT

Modern deep neural networks (DNNs) become frail when the datasets contain noisy (incorrect) class labels. Robust techniques in the presence of noisy labels can be categorized into two folds: developing noise-robust functions or using noise-cleansing methods by detecting the noisy data. Recently, noise-cleansing methods have been considered as the most competitive noisy-label learning algorithms. Despite their success, their noisy label detectors are often based on heuristics more than a theory, requiring a robust classifier to predict the noisy data with loss values. In this paper, we propose a novel detector for filtering label noise. Unlike most existing methods, we focus on each data's latent representation dynamics and measure the alignment between the latent distribution and each representation using the eigendecomposition of the data gram matrix. Our framework, coined as filtering noisy instances via their eigenvectors (FINE), provides a robust detector with derivative-free simple methods having theoretical guarantees. Under our framework, we propose three applications of the FINE: sample-selection approach, semi-supervised learning approach, and collaboration with noise-robust loss functions. Experimental results show that the proposed methods consistently outperform corresponding baselines for all three applications on various benchmark datasets.

研究动机与目标

• Motivate robust learning from datasets with noisy labels where traditional loss-based detectors fail due to classifier bias.
• Develop a noise detector that relies on latent representation geometry rather than posterior predictions.
• Provide a theoretically grounded framework with guarantees for filtering noisy instances.
• Demonstrate the detector across three LNL applications: sample selection, semi-supervised learning, and collaboration with noise-robust losses.

提出的方法

• 从预前（pre-logit）表示构建各类别的 Gram 矩阵，并进行特征分解以获得每个类别的第一特征向量。
• 将数据点的对齐度定义为与该类别第一特征向量的平方内积，并用高斯混合模型（Gaussian Mixture Model）来建模对齐分布，以将干净样本与噪声样本分离。
• 通过选择在 GMM 的“干净分量”内的对齐分数来过滤噪声数据，而无需估计噪声率。
• 给出对估计的干净特征向量在有标签噪声下的扰动的理论界限（定理 1），将扰动与噪声比及干净/有噪声类别方向之间的夹角联系起来。
• 通过用少量数据近似特征向量来展示可扩展性，保持高精度。
• 将 FINE 集成到三个 LNL 范式中：（1）样本选择（替换现有检测器），（2）SSL（替换基于损失的筛选），（3）与对噪声鲁棒的损失协作。

实验结果

研究问题

• RQ1 能否通过潜在空间的特征向量结构提供一个鲁棒的、无需估计噪声率的无导数检测器以检测有噪声标签？
• RQ2 对齐到第一特征向量在不同数据集和噪声模式下将干净样本与噪声样本分离得有多好？
• RQ3 与现有检测器相比，基于 FINE 的检测器在样本选择、SSL 和鲁棒损失协作中是否提高了性能？
• RQ4 在现实假设下，哪些理论保证可以将检测器的特征向量对齐与干净数据识别联系起来？

主要发现

• FINE 在 CIFAR-10/100 的对称噪声和非对称噪声的多种设定下，在样本选择任务上始终优于竞争基线。
• 用 FINE 替代基于损失的筛选，在与 Co-teaching 变体（F-Co-teaching）以及 TopoFilter/CRUST 基线的整合中获得显著提升。
• 在 SSL 中，FINE 增强的 DivideMix（F-DivideMix）在测试精度方面高于 DivideMix，并在严重噪声下与领先的 SSL 方法相竞争。
• FINE 指导下与对噪声鲁棒的损失（GCE、SCE、ELR）协作可在高噪声情形下提升泛化。
• 对 Clothing1M 的实验显示竞争性能，表明 FINE 在现实世界中的适用性不仅限于合成基准。