[Paper Review] Are Transformers More Robust Than CNNs?

Yutong Bai, Jieru Mei+2 more|arXiv (Cornell University)|Nov 10, 2021

Adversarial Robustness in Machine LearningComputer Science93 citations

TL;DR

With a unified training setup, CNNs and Transformers show similar adversarial robustness, while Transformers’ self-attention architecture improves robustness to out-of-distribution data.

ABSTRACT

Transformer emerges as a powerful tool for visual recognition. In addition to demonstrating competitive performance on a broad range of visual benchmarks, recent works also argue that Transformers are much more robust than Convolutions Neural Networks (CNNs). Nonetheless, surprisingly, we find these conclusions are drawn from unfair experimental settings, where Transformers and CNNs are compared at different scales and are applied with distinct training frameworks. In this paper, we aim to provide the first fair & in-depth comparisons between Transformers and CNNs, focusing on robustness evaluations. With our unified training setup, we first challenge the previous belief that Transformers outshine CNNs when measuring adversarial robustness. More surprisingly, we find CNNs can easily be as robust as Transformers on defending against adversarial attacks, if they properly adopt Transformers' training recipes. While regarding generalization on out-of-distribution samples, we show pre-training on (external) large-scale datasets is not a fundamental request for enabling Transformers to achieve better performance than CNNs. Moreover, our ablations suggest such stronger generalization is largely benefited by the Transformer's self-attention-like architectures per se, rather than by other training setups. We hope this work can help the community better understand and benchmark the robustness of Transformers and CNNs. The code and models are publicly available at https://github.com/ytongbai/ViTs-vs-CNNs.

Motivation & Objective

• Motivate a fair benchmark comparing CNNs and Transformers in robustness rather than biased, scale-disparate evaluations
• Evaluate adversarial robustness under perturbation-based and patch-based attacks
• Assess robustness generalization on out-of-distribution benchmarks (ImageNet-A, ImageNet-C, Stylized-ImageNet)
• Investigate how training recipes and architectures influence robustness conclusions
• Provide actionable guidance for benchmarking robustness across architectures

Proposed method

• Train ResNet-50 (CNN) and DeiT-S (Transformer) with aligned model capacity (~22–25M parameters)
• Use 100-epoch training with standard CNN and 100-epoch DeiT-S training following DeiT recipe
• Evaluate adversarial robustness against AutoAttack, PGD variants, and Texture Patch Attack (TPA)
• Evaluate robustness on out-of-distribution datasets: ImageNet-A, ImageNet-C, Stylized-ImageNet
• Conduct ablations: optimizers, learning rate schedulers, augmentation strategies (RandAug, Mixup, CutMix), activation functions (ReLU vs GELU)
• Explore distillation and hybrid architectures to test robustness transfer across architectures

Experimental results

Research questions

• RQ1 Do Transformers inherently outperform CNNs in adversarial robustness under fair, aligned training settings?
• RQ2 Does adopting Transformers’ training recipes to CNNs improve their adversarial robustness?
• RQ3 Do Transformers provide superior robustness to out-of-distribution data independent of pre-training scale?
• RQ4 Is the robustness advantage of Transformers primarily due to their self-attention-like architecture or to training procedures?

Key findings

• Under unified training conditions, Transformers are not more robust than CNNs to perturbation-based adversarial attacks.
• Adversarial training with CNN-style recipes can match DeiT-S robustness when using GELU, but may reduce clean accuracy compared to standard CNN training.
• CNNs can significantly improve patch-based attack robustness with strong augmentations (e.g., CutMix), approaching or surpassing Transformers in some settings.
• Transformers consistently outperform CNNs on out-of-distribution generalization benchmarks (ImageNet-A, ImageNet-C, Stylized-ImageNet) even without large external pre-training.
• The robustness gap on out-of-distribution data is largely attributed to architecture (self-attention) rather than training setup alone, as hybrid models with Transformer blocks show improved robustness over CNNs.
• Knowledge distillation transfers robustness: DeiT-S as teacher improves student ResNet-50 but the reverse (ResNet-50 as teacher) does not yield the same robustness gains, suggesting architectural robustness is not easily transferred via distillation.
• Scaling comparisons across model sizes (ResNet-18/50/101 vs DeiT-Mini/S) consistently show Transformers offer stronger robustness generalization to out-of-distribution data.