[Paper Review] Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Models

Naman D Singh, Francesco Croce+1 more|arXiv (Cornell University)|Mar 3, 2023

Adversarial Robustness in Machine Learning13 citations

TL;DR

The paper studies how architecture (ViT, ConvNeXt, ConvStem) and training schemes affect adversarial robustness on ImageNet, showing ConvStem and strong pre-training with heavy augmentation yield state-of-the-art l_infinity robustness and better generalization to unseen threat models.

ABSTRACT

While adversarial training has been extensively studied for ResNet architectures and low resolution datasets like CIFAR, much less is known for ImageNet. Given the recent debate about whether transformers are more robust than convnets, we revisit adversarial training on ImageNet comparing ViTs and ConvNeXts. Extensive experiments show that minor changes in architecture, most notably replacing PatchStem with ConvStem, and training scheme have a significant impact on the achieved robustness. These changes not only increase robustness in the seen $\ell_\infty$-threat model, but even more so improve generalization to unseen $\ell_1/\ell_2$-attacks. Our modified ConvNeXt, ConvNeXt + ConvStem, yields the most robust $\ell_\infty$-models across different ranges of model parameters and FLOPs, while our ViT + ConvStem yields the best generalization to unseen threat models.

Motivation & Objective

• Investigate how architecture choices influence robustness to seen and unseen adversarial attacks on ImageNet.
• Evaluate ViT, ConvNeXt, and isotropic ConvNeXt with and without ConvStem.
• Assess the impact of strong pre-training and heavy data augmentation on robustness.
• Examine test-time resolution effects and optimization schemes on robustness.
• Provide practical training recipes achieving high l_infinity robustness on ImageNet.

Proposed method

• Compare ViT and ConvNeXt family architectures on ImageNet under l_infinity adversarial training.
• Replace PatchStem with ConvStem to create CvSt variants and evaluate robustness.
• Use strong clean pre-training to initialize adversarial training.
• Apply heavy data augmentation (RandAugment, MixUp, CutMix, Random Erasing) during AT.
• Evaluate robustness using AutoAttack across l_infinity, l2, and l1 threat models.
• Analyze test-time resolution effects and fine-tuning to larger radii.

Experimental results

Research questions

• RQ1 How do architectural components (PatchStem vs ConvStem) affect robustness to seen and unseen threat models on ImageNet?
• RQ2 What is the impact of strong pre-training and heavy data augmentation on l_infinity robustness and generalization to l1/l2 attacks?
• RQ3 Does increasing test-time resolution improve robust performance without sacrificing robustness to stronger perturbations?
• RQ4 Can ConvStem-enabled ConvNeXt and ViT models outperform existing SOTA in l_infinity robustness on ImageNet across model sizes?
• RQ5 How do 1-step vs 2-step adversarial training affect robustness and training efficiency for these architectures?

Key findings

• ConvStem consistently improves l_infinity robustness and clean accuracy across isotropic ConvNeXt and ViT architectures.
• ConvStem + strong pre-training + heavy augmentation yields large gains in unseen l1 and l2 robustness across architectures.
• ConvNeXt-T + ConvStem achieves 50.2% l_infinity robust accuracy at epsilon=4/255, outperforming prior state-of-the-art by 5.8% for small models and 2.8% for large models.
• ViT + ConvStem provides best generalization to unseen threat models among evaluated architectures.
• Higher test-time resolution can improve robust accuracy for several top models, despite stronger attacks at fixed radii.
• Training with 2-step APGD AT yields competitive robustness with lower cost than more steps; 50-epoch training with 2-step AT can beat some longer training regimes.