[论文解读] Robustness of different loss functions and their impact on networks learning capability

Vishal Rajput|arXiv (Cornell University)|Oct 15, 2021

Adversarial Robustness in Machine Learning被引用 6

一句话总结

本文研究了各种损失函数（特别是广义损失，如二元交叉熵）和专用损失（如Dice损失、焦点损失）在深度学习模型中的鲁棒性。通过在对抗性鲁棒性方面进行受控实验，表明结合损失函数可增强模型对基于梯度扰动的抵抗能力，混合损失策略在显著梯度攻击下能更好地保持准确率。

ABSTRACT

Recent developments in AI have made it ubiquitous, every industry is trying to adopt some form of intelligent processing of their data. Despite so many advances in the field, AIs full capability is yet to be exploited by the industry. Industries that involve some risk factors still remain cautious about the usage of AI due to the lack of trust in such autonomous systems. Present-day AI might be very good in a lot of things but it is very bad in reasoning and this behavior of AI can lead to catastrophic results. Autonomous cars crashing into a person or a drone getting stuck in a tree are a few examples where AI decisions lead to catastrophic results. To develop insight and generate an explanation about the learning capability of AI, we will try to analyze the working of loss functions. For our case, we will use two sets of loss functions, generalized loss functions like Binary cross-entropy or BCE and specialized loss functions like Dice loss or focal loss. Through a series of experiments, we will establish whether combining different loss functions is better than using a single loss function and if yes, then what is the reason behind it. In order to establish the difference between generalized loss and specialized losses, we will train several models using the above-mentioned losses and then compare their robustness on adversarial examples. In particular, we will look at how fast the accuracy of different models decreases when we change the pixels corresponding to the most salient gradients.

研究动机与目标

• 理解不同损失函数如何影响深度神经网络的学习能力和鲁棒性。
• 评估结合多个损失函数是否相比单损失训练能提升模型性能和鲁棒性。
• 研究在对抗样本下人工智能系统的失效模式，特别是在安全关键应用中。
• 为专用损失函数在数据不平衡或对抗性场景下表现优于通用损失函数的原因提供实证洞察。

提出的方法

• 使用广义损失（如二元交叉熵）和专用损失（如Dice损失、焦点损失）的组合训练多个深度学习模型。
• 通过修改对应于最显著梯度的像素，应用基于梯度的对抗攻击。
• 通过测量在扰动强度逐渐增加时准确率下降的速率来评估鲁棒性。
• 使用标准化基准比较不同损失配置下的模型性能。
• 通过在图像分类任务中对类别不平衡进行受控消融研究来评估鲁棒性。
• 跟踪沿高梯度方向施加扰动时准确率随时间的下降情况。

实验结果

研究问题

• RQ1 损失函数的选择如何影响深度神经网络在对抗攻击下的鲁棒性？
• RQ2 结合多个损失函数是否相比单损失训练能带来更好的泛化能力和鲁棒性？
• RQ3 为何专用损失函数（如焦点损失或Dice损失）在类别不平衡或对抗性场景下表现更优？
• RQ4 当沿最显著梯度方向施加扰动时，模型准确率下降的速度如何？
• RQ5 损失函数设计与网络在分布偏移下推理能力之间有何关系？

主要发现

• 使用混合损失函数（如BCE + Dice或BCE + 焦点损失）训练的模型在对抗攻击下表现出显著更慢的准确率下降速度，相比单损失模型。
• 广义损失与专用损失的结合提升了鲁棒性，尤其在类别不平衡的情境下。
• 焦点损失在处理长尾数据分布时表现更优，能在扰动下维持更高准确率。
• 仅使用二元交叉熵的模型在遭受基于梯度的对抗攻击时表现出更快的准确率下降。
• 最显著的梯度被发现对确定模型失效点具有高度影响力，尤其是在单损失设置中。
• 鲁棒性并非仅取决于模型架构，损失函数设计和训练目标也具有显著影响。