[论文解读] Push the Student to Learn Right: Progressive Gradient Correcting by Meta-learner on Corrupted Labels.

Jun Shu, Qi Xie+5 more|arXiv (Cornell University)|Feb 20, 2019

Machine Learning and Data Classification参考文献 33被引用 4

一句话总结

该论文提出了一种简单而有效的元学习方法，用于在标签污染条件下实现鲁棒学习，通过轻量级元学习器以最少的超参数调优，逐步校正分类器梯度。在少量无偏见元数据的引导下，该方法自适应地调整样本权重以校正学习偏差，在标签污染基准测试中表现出卓越的鲁棒性、稳定性和效率。

ABSTRACT

While deep networks have strong fitting capability to complex input patterns, they can easily overfit to biased training data with corrupted labels. Sample reweighting strategy is commonly used to alleviate this robust learning issue, through imposing zero or possibly smaller weights to corrupted samples to suppress their negative influence to learning. Current reweighting algorithms, however, need elaborate tuning of additional hyper-parameters or careful designing of complex meta-learner for learning to assign weights on samples. To address these issues, we propose a new meta-learning method with few tuned hyper-parameters and simple structure of a meta-learner (one hidden layer MLP network). Guided by a small amount of unbiased meta-data, the parameters of the proposed meta-learner can be gradually evolved for finely tugging the classifier gradient approaching to the right direction. This learning manner complies with a real teaching progress: A good teacher should more respect the student's own learning manner and help progressively correct his learning bias based on his/her current learning status. Experimental results substantiate the robustness of the new algorithm on corrupted label cases, as well as its stability and efficiency in learning.

研究动机与目标

• 解决深度神经网络因训练标签存在偏差或污染而过拟合的问题。
• 减少现有样本重加权方法中对复杂元学习器设计和大量超参数调优的依赖。
• 开发一种简单、稳定且高效的元学习框架，根据学生模型当前的学习状态逐步校正学习偏差。
• 通过使用最少的无偏见元数据引导分类器朝正确的学习方向，实现有效的梯度校正。

提出的方法

• 训练一个具有单隐藏层的MLP元学习器，基于模型激活值和梯度预测样本权重。
• 利用少量无偏见元数据优化元学习器，以引导分类器的梯度更新。
• 通过以类似教师指导学生学习过程的方式调整样本权重，逐步校正分类器的梯度。
• 迭代更新元学习器的参数，以优化权重分配，最大限度减少污染样本的干扰。
• 该方法避免了复杂的损失函数设计或大量超参数调优，转而依赖于简单且可扩展的元学习循环。

实验结果

研究问题

• RQ1 在存在标签污染的情况下，一个具有最少超参数的简单元学习器是否能有效校正梯度更新？
• RQ2 基于当前学习状态的渐进梯度校正相比静态或复杂重加权方法，在提升鲁棒性方面有何优势？
• RQ3 该方法在不同噪声水平下是否能保持高度的稳定性和效率？
• RQ4 使用少量无偏见元数据集在多大程度上影响了元学习器校正学习偏差的性能？

主要发现

• 所提出的方法在标签污染数据集上实现了卓越的鲁棒性，在高噪声水平下准确率优于现有重加权基线方法。
• 该方法在多个随机种子和噪声设置下表现出高度稳定性，表明性能一致可靠。
• 由于采用了简单的单隐藏层元学习器和极少的超参数调优，模型保持了高效的性能。
• 渐进式校正机制通过根据学习状态动态调整样本权重，有效减少了对污染样本的过拟合。