[论文解读] Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better

Bojia Zi, Shihao Zhao+2 more|arXiv (Cornell University)|Aug 18, 2021

Adversarial Robustness in Machine Learning参考文献 61被引用 5

一句话总结

该论文提出了一种名为鲁棒软标签对抗蒸馏（RSLAD）的知识蒸馏方法，通过利用经过对抗训练的大规模教师模型生成的鲁棒软标签（RSLs），提升小型学生模型的对抗鲁棒性。RSLAD在所有损失项中均使用RSLs，相较于现有方法，在AutoAttack攻击下于CIFAR-10和CIFAR-100数据集上实现了最先进（SOTA）的鲁棒性表现，其中使用WideResNet-34-10作为教师的ResNet-18学生模型甚至超越了其教师模型。

ABSTRACT

Adversarial training is one effective approach for training robust deep neural networks against adversarial attacks. While being able to bring reliable robustness, adversarial training (AT) methods in general favor high capacity models, i.e., the larger the model the better the robustness. This tends to limit their effectiveness on small models, which are more preferable in scenarios where storage or computing resources are very limited (e.g., mobile devices). In this paper, we leverage the concept of knowledge distillation to improve the robustness of small models by distilling from adversarially trained large models. We first revisit several state-of-the-art AT methods from a distillation perspective and identify one common technique that can lead to improved robustness: the use of robust soft labels -- predictions of a robust model. Following this observation, we propose a novel adversarial robustness distillation method called Robust Soft Label Adversarial Distillation (RSLAD) to train robust small student models. RSLAD fully exploits the robust soft labels produced by a robust (adversarially-trained) large teacher model to guide the student's learning on both natural and adversarial examples in all loss terms. We empirically demonstrate the effectiveness of our RSLAD approach over existing adversarial training and distillation methods in improving the robustness of small models against state-of-the-art attacks including the AutoAttack. We also provide a set of understandings on our RSLAD and the importance of robust soft labels for adversarial robustness distillation.

研究动机与目标

• 在移动设备等资源受限环境中，提升小型轻量级深度神经网络的对抗鲁棒性。
• 探究软标签在对抗鲁棒性蒸馏中的作用，特别是区分鲁棒软标签与非鲁棒软标签的差异。
• 开发一种新型蒸馏框架，充分挖掘鲁棒软标签以提升学生模型性能。
• 理解教师与学生模型复杂度之间的相互作用及其对蒸馏效果的影响。
• 通过实证验证，鲁棒软标签对于有效对抗鲁棒性蒸馏至关重要。

提出的方法

• RSLAD在学生模型训练的所有损失项中，用鲁棒软标签（RSLs）替代硬标签，包括交叉熵损失和一致性损失。
• RSLs由一个大型、经过对抗训练的教师模型（例如使用TRADES或PGD-based AT）生成，确保其反映鲁棒预测结果。
• 学生模型通过RSLs作为监督信号，模仿教师模型在自然样本和对抗样本上的预测结果。
• 该方法在所有训练信号——干净数据、对抗数据和一致性——上应用知识蒸馏，以RSLs作为主要监督信号。
• 该框架在CIFAR-10和CIFAR-100数据集上进行评估，采用ResNet-18和WideResNet作为学生模型，并使用多种教师架构。
• 消融研究对比了RSLs与自然软标签（NSLs）和光滑软标签（SSLs），证明了软标签鲁棒性在其中的关键作用。

Figure 1: An overview of the proposed RSLAD framework, in comparison with 4 existing methods including TRADES , MART , ARD and IAD . Black solid arrows represent training with hard labels $y$ ; yellow and blue dashed arrows represent predicting process for natural and adversarial examples respective

实验结果

研究问题

• RQ1 使用来自对抗训练教师模型的鲁棒软标签，是否能显著提升小型学生模型的对抗鲁棒性？
• RQ2 软标签类型的选择（RSL vs. NSL vs. SSL）如何影响蒸馏后学生模型的鲁棒性？
• RQ3 教师模型复杂度对蒸馏后学生模型鲁棒性的影响是什么？
• RQ4 从一个中等规模的教师模型蒸馏，是否能获得比从更大或相同规模教师模型蒸馏更好的鲁棒性？
• RQ5 在对抗蒸馏中是否存在鲁棒性饱和或鲁棒性欠拟合现象，其对模型性能有何影响？

主要发现

• RSLAD在CIFAR-10上实现了最先进（SOTA）的鲁棒性表现，当使用WideResNet-34-10作为教师时，ResNet-18学生模型在AutoAttack攻击下达到51.49%的准确率。
• 使用非鲁棒软标签（如自然软标签NSLs或光滑软标签SSLs）会导致鲁棒性显著下降，其中NSLs使AA准确率降至42.08%。
• 学生模型的鲁棒性并非随教师复杂度单调提升；相反，观察到一种‘鲁棒性饱和’现象，即性能在中等规模教师（如WideResNet-34-10）处达到峰值。
• 令人惊讶的是，当从更宽、更复杂的教师模型蒸馏时，ResNet-18学生模型的鲁棒性甚至超过了其自身教师模型，表明对抗训练中存在鲁棒性欠拟合现象。
• 当教师模型规模约为学生模型的4.5倍时（例如，使用WideResNet-34-10蒸馏ResNet-18），可获得最佳鲁棒性增益，证实了教师-学生容量平衡的重要性。
• 实证结果表明，鲁棒软标签对于有效蒸馏至关重要，因为它们编码了教师模型的真实鲁棒行为，而与之相比，非鲁棒标签会误导学生模型。

Figure 2: Attention and saliency maps on adversarial examples. Teacher: WideResNet-34-10 trained by TRADES; ARD: the ResNet-18 student trained using ARD with the Teacher network; RSLAD: the ResNet-18 student trained using our RSLAD with the Teacher network. Heatmaps are generated by Grad-Cam [ 38 ]