[论文解读] A Light Recipe to Train Robust Vision Transformers

Edoardo Debenedetti, Vikash Sehwag+1 more|arXiv (Cornell University)|Sep 15, 2022

Adversarial Robustness in Machine Learning被引用 4

一句话总结

本文提出了一种轻量级训练方案，通过用更强的权重衰减和epsilon热身（epsilon-warmup）替代繁重的数据增强，显著提升了视觉Transformer（ViTs）的对抗鲁棒性，在ImageNet-1k和下游任务上实现了最先进（SOTA）的鲁棒准确率，甚至在对抗训练下超越了ResNet-50的性能。

ABSTRACT

In this paper, we ask whether Vision Transformers (ViTs) can serve as an underlying architecture for improving the adversarial robustness of machine learning models against evasion attacks. While earlier works have focused on improving Convolutional Neural Networks, we show that also ViTs are highly suitable for adversarial training to achieve competitive performance. We achieve this objective using a custom adversarial training recipe, discovered using rigorous ablation studies on a subset of the ImageNet dataset. The canonical training recipe for ViTs recommends strong data augmentation, in part to compensate for the lack of vision inductive bias of attention modules, when compared to convolutions. We show that this recipe achieves suboptimal performance when used for adversarial training. In contrast, we find that omitting all heavy data augmentation, and adding some additional bag-of-tricks ($\varepsilon$-warmup and larger weight decay), significantly boosts the performance of robust ViTs. We show that our recipe generalizes to different classes of ViT architectures and large-scale models on full ImageNet-1k. Additionally, investigating the reasons for the robustness of our models, we show that it is easier to generate strong attacks during training when using our recipe and that this leads to better robustness at test time. Finally, we further study one consequence of adversarial training by proposing a way to quantify the semantic nature of adversarial perturbations and highlight its correlation with the robustness of the model. Overall, we recommend that the community should avoid translating the canonical training recipes in ViTs to robust training and rethink common training choices in the context of adversarial training.

研究动机与目标

• 探究视觉Transformer（ViTs）在对抗训练下是否能实现优于卷积神经网络（CNNs）的对抗鲁棒性。
• 识别一种专为ViT对抗训练量身定制的训练方案，区别于标准ViT训练方案。
• 理解为何标准ViT训练方案——尤其是繁重的数据增强——在对抗训练中会损害鲁棒性。
• 量化对抗扰动的语义特性，并将其与模型鲁棒性关联。
• 证明所提出的方案在不同ViT架构之间具有泛化能力，并支持在小样本数据集上高效微调。

提出的方法

• 在ImageNet-1k的一个子集上进行系统性消融研究，评估数据增强、权重衰减和对抗训练超参数对ViT鲁棒性的影响。
• 将强数据增强（如mixup、CutMix）替换为最小化增强，以提升鲁棒性，这与标准ViT训练实践相反。
• 在对抗训练中引入epsilon-warmup，逐步增加扰动幅度，以稳定训练过程。
• 应用更大的权重衰减值以提升泛化能力和鲁棒性，尤其在缺乏强数据增强的情况下。
• 以Cross-Covariance Image Transformer（XCiT）为主要架构验证该方案，随后扩展至其他ViT变体和ConvNeXt。
• 通过梯度可视化和语义对齐度量来量化对抗扰动，分析其感知合理性。

实验结果

研究问题

• RQ1 经过自定义对抗训练方案微调的ViT是否在对抗鲁棒性上优于标准CNN？
• RQ2 为何在ViT训练中常见的繁重数据增强会降低对抗训练中的性能？
• RQ3 是否可通过简化数据增强策略并结合超参数调优（如权重衰减、epsilon-warmup）显著提升ViT鲁棒性？
• RQ4 鲁棒ViT与ResNet等标准模型在对抗扰动的语义特性上存在哪些差异？
• RQ5 所提出的方案是否在不同ViT架构及下游微调任务中均具有泛化能力？

主要发现

• 与标准ViT训练方案相比，该方案将ImageNet-1k上的鲁棒准确率提升了13.1%。
• 该方案在ImageNet-1k上实现了最先进（SOTA）的鲁棒准确率，并在下游微调任务（如CIFAR-100、Caltech-101）上最高提升了5.5%的鲁棒准确率。
• 在省略繁重数据增强的同时，提升权重衰减并使用epsilon-warmup，可在不损失干净准确率的前提下显著提升鲁棒性。
• 采用新方案训练的模型在训练过程中表现出更有效的1步攻击，表明其学习到了更强的对抗信号。
• 鲁棒XCiT模型的对抗扰动比ResNet的扰动更具语义一致性，与人类感知更吻合，表明其通过更具意义的扰动实现了更高鲁棒性。
• 该方案可泛化至其他ViT架构，甚至适用于ConvNeXt，仅作微小调整即可实现SOTA性能。