[论文解读] Exploring the Relationship between Architecture and Adversarially Robust Generalization

Aishan Liu, Shiyu Tang+5 more|arXiv (Cornell University)|Sep 28, 2022

Adversarial Robustness in Machine Learning被引用 5

一句话总结

本文研究了神经网络架构与对抗性鲁棒泛化之间的关系，发现与CNN相比，视觉Transformer（如PVT、CoAtNet）在多种$ε$-范数攻击下泛化能力更强，而CNN往往对特定攻击过拟合。作者通过Rademacher复杂度分析，理论上将这一现象归因于注意力机制中更高的权重稀疏性，从而增强了鲁棒性。

ABSTRACT

Adversarial training has been demonstrated to be one of the most effective remedies for defending adversarial examples, yet it often suffers from the huge robustness generalization gap on unseen testing adversaries, deemed as the adversarially robust generalization problem. Despite the preliminary understandings devoted to adversarially robust generalization, little is known from the architectural perspective. To bridge the gap, this paper for the first time systematically investigated the relationship between adversarially robust generalization and architectural design. Inparticular, we comprehensively evaluated 20 most representative adversarially trained architectures on ImageNette and CIFAR-10 datasets towards multiple `p-norm adversarial attacks. Based on the extensive experiments, we found that, under aligned settings, Vision Transformers (e.g., PVT, CoAtNet) often yield better adversarially robust generalization while CNNs tend to overfit on specific attacks and fail to generalize on multiple adversaries. To better understand the nature behind it, we conduct theoretical analysis via the lens of Rademacher complexity. We revealed the fact that the higher weight sparsity contributes significantly towards the better adversarially robust generalization of Transformers, which can be often achieved by the specially-designed attention blocks. We hope our paper could help to better understand the mechanism for designing robust DNNs. Our model weights can be found at http://robust.art.

研究动机与目标

• 系统研究架构设计对对抗性鲁棒泛化的影响，特别是针对多种$ε$-范数攻击的影响。
• 识别在对抗性鲁棒性训练下促进更好泛化的架构因素。
• 解释尽管训练设置相似，为何视觉Transformer在未见对抗攻击下泛化优于CNN。
• 利用Rademacher复杂度对权重稀疏性在对抗性鲁棒泛化中的作用进行理论分析。

提出的方法

• 在ImageNette和CIFAR-10数据集上，对20种对抗性训练的架构（包括ViT、PVT、ResNet、VGG）进行多种$ε$-范数攻击（$\ell_1$、$\ell_2$、$\ell_\infty$）下的全面评估。
• 所有模型在对齐的超参数下进行训练，以确保架构间比较的公平性。
• 利用Rademacher复杂度进行理论分析，将权重稀疏性与Transformer中鲁棒泛化的提升联系起来。
• 通过CIFAR-10-C和ImageNette-C评估在常见噪声下的泛化能力，以评估模型在对抗样本之外的鲁棒性。
• 对patch大小和架构组件（如注意力模块）进行消融研究，以隔离其对鲁棒性的影响。

实验结果

研究问题

• RQ1 在对抗性训练下，视觉Transformer是否在多种$ε$-范数对抗攻击中均优于CNN实现更好的泛化？
• RQ2 哪些架构特性有助于实现更好的对抗性鲁棒泛化？
• RQ3 权重稀疏性如何影响对抗性鲁棒泛化？它是否是Transformer中关键因素？
• RQ4 为何CNN往往对特定攻击过拟合，而Transformer在多种攻击类型中泛化能力更优？
• RQ5 Rademacher复杂度能否用于理论解释不同架构的鲁棒泛化行为？

主要发现

• 视觉Transformer如PVT和CoAtNet在多种$ε$-范数攻击下表现出显著更高的最差情况鲁棒准确率（例如，PVTv2在ImageNette上使用patch size 4时最差准确率为33.54%），优于CNN。
• CNN如ResNet和VGG对特定攻击表现出强烈过拟合，例如ViTAE在$\ell_1$攻击下最差准确率低至21.10%。
• Transformer在常见噪声下表现出更优的泛化能力：例如，CoAtNet在CIFAR-10-C上达到79.91%准确率，优于ResNet（干净数据81.20%，CIFAR-10-C 79.24%）的噪声泛化表现。
• 理论分析表明，更高的权重稀疏性——通常通过注意力模块实现——能显著提升对抗性鲁棒泛化能力，该结论通过Rademacher复杂度界得到形式化证明。
• PVTv2在patch size 1时在ImageNette上达到94.60%原始准确率和54.59% $\ell_\infty$-PGD鲁棒准确率，表明其在多种攻击类型下均具备强大的鲁棒泛化能力。
• 在ImageNet上，Transformer的干净准确率与鲁棒准确率之间的差距大于CNN，表明架构设计必须针对特定的鲁棒性与泛化权衡进行定制。