[论文解读] Improving MAE against CCE under Label Noise.

Xinshao Wang, Elyor Kodirov+2 more|arXiv (Cornell University)|Mar 28, 2019

Machine Learning and Data Classification参考文献 26被引用 11

一句话总结

本文提出改进平均绝对误差（IMAE），一种抗噪声损失函数，通过控制训练样本间梯度幅值的差异性，缓解了标准MAE在高标签噪声下的欠拟合行为。IMAE在保持MAE理论抗噪性的同时，在图像分类与视频检索任务中，于合成与真实噪声标签下，显著优于交叉熵损失（CCE）与标准MAE。

ABSTRACT

Label noise is inherent in many deep learning tasks when the training set becomes large. A typical approach to tackle noisy labels is using robust loss functions. Categorical cross entropy (CCE) is a successful loss function in many applications. However, CCE is also notorious for fitting samples with corrupted labels easily. In contrast, mean absolute error (MAE) is noise-tolerant theoretically, but it generally works much worse than CCE in practice. In this work, we have three main points. First, to explain why MAE generally performs much worse than CCE, we introduce a new understanding of them fundamentally by exposing their intrinsic sample weighting schemes from the perspective of every sample's gradient magnitude with respect to logit vector. Consequently, we find that MAE's differentiation degree over training examples is too small so that informative ones cannot contribute enough against the non-informative during training. Therefore, MAE generally underfits training data when noise rate is high. Second, based on our finding, we propose an improved MAE (IMAE), which inherits MAE's good noise-robustness. Moreover, the differentiation degree over training data points is controllable so that IMAE addresses the underfitting problem of MAE. Third, the effectiveness of IMAE against CCE and MAE is evaluated empirically with extensive experiments, which focus on image classification under synthetic corrupted labels and video retrieval under real noisy labels.

研究动机与目标

• 解释为何尽管MAE在理论上对标签噪声具有鲁棒性，但在实际中仍表现不如CCE。
• 识别MAE在高噪声率下出现欠拟合的根本原因，即训练样本间梯度幅值差异性不足。
• 设计一种新损失函数，在保留MAE抗噪性的同时，增强信息性样本的贡献。
• 在合成与真实噪声标签下，通过图像分类与视频检索任务，对所提方法与CCE及标准MAE进行实证验证。

提出的方法

• 通过分析每个样本损失相对于logit向量的梯度幅值，对MAE与CCE进行新的理论分析。
• 揭示MAE在训练样本间梯度幅值差异性较低，导致在高噪声下出现欠拟合。
• 提出IMAE，一种改进的MAE损失函数，引入可学习缩放机制以控制梯度幅值的差异性程度。
• 在IMAE中采用样本级加权策略，增强信息性样本相对于噪声样本的贡献。
• 使用可微分的温度参数控制梯度幅值分布的锐度，实现对差异性的可控调节。
• 在图像与视频基准上采用标准训练协议，并与交叉熵和MAE基线进行公平比较。

实验结果

研究问题

• RQ1 为何MAE在高标签噪声下仍出现欠拟合，尽管其在理论上对噪声具有鲁棒性？
• RQ2 MAE在训练样本间的梯度幅值分布与CCE有何不同，这种差异对模型性能有何影响？
• RQ3 我们能否通过增强梯度幅值的差异性来提升MAE性能，同时不牺牲其抗噪性？
• RQ4 在合成与真实噪声标签下，所提出的IMAE损失与CCE及标准MAE相比，在准确率与鲁棒性方面表现如何？

主要发现

• MAE在高噪声下表现不佳，因其梯度幅值在训练样本间差异性不足，导致信息性样本被过度弱化。
• 在合成标签噪声下的图像分类任务中，IMAE相比标准MAE实现了显著更高的准确率，尤其在高噪声率下优势明显。
• 在真实世界噪声视频检索基准上，IMAE优于CCE，展现出在实际场景中的更强鲁棒性。
• IMAE在保持与MAE相当的强抗噪性的同时，在干净数据上的性能接近或超过CCE。
• 消融实验确认，控制梯度幅值差异性对于提升MAE在噪声标签下的泛化能力至关重要。
• IMAE中的温度控制缩放机制实现了鲁棒性与性能之间的有效权衡，使其能适应不同噪声水平。