[Paper Review] Smooth Adversarial Training

Cihang Xie, Mingxing Tan+3 more|arXiv (Cornell University)|Jun 25, 2020

Adversarial Robustness in Machine Learning82 references99 citations

TL;DR

SAT replaces ReLU with smooth activations to strengthen adversarial training, yielding higher robustness without sacrificing accuracy or extra computation, demonstrated on ImageNet with ResNet-50 and EfficientNet-L1.

ABSTRACT

It is commonly believed that networks cannot be both accurate and robust, that gaining robustness means losing accuracy. It is also generally believed that, unless making networks larger, network architectural elements would otherwise matter little in improving adversarial robustness. Here we present evidence to challenge these common beliefs by a careful study about adversarial training. Our key observation is that the widely-used ReLU activation function significantly weakens adversarial training due to its non-smooth nature. Hence we propose smooth adversarial training (SAT), in which we replace ReLU with its smooth approximations to strengthen adversarial training. The purpose of smooth activation functions in SAT is to allow it to find harder adversarial examples and compute better gradient updates during adversarial training. Compared to standard adversarial training, SAT improves adversarial robustness for "free", i.e., no drop in accuracy and no increase in computational cost. For example, without introducing additional computations, SAT significantly enhances ResNet-50's robustness from 33.0% to 42.3%, while also improving accuracy by 0.9% on ImageNet. SAT also works well with larger networks: it helps EfficientNet-L1 to achieve 82.2% accuracy and 58.6% robustness on ImageNet, outperforming the previous state-of-the-art defense by 9.5% for accuracy and 11.6% for robustness. Models are available at https://github.com/cihangxie/SmoothAdversarialTraining.

Motivation & Objective

• Challenge the belief that robustness requires sacrificing accuracy or larger models.
• Investigate how activation smoothness affects adversarial training quality.
• Propose and evaluate Smooth Adversarial Training (SAT) using smooth activations.
• Quantify gains in robustness and accuracy across large-scale and scalable architectures.

Proposed method

• Identify ReLU as a source of weak adversarial training due to non-smooth gradients.
• Introduce smooth activation functions (e.g., Softplus, SILU, GELU, ELU variants) for both forward and backward passes in SAT.
• Train models with adversarial examples using PGD-based attacks, comparing gradient quality in attacker and optimizer.
• Evaluate SAT on ImageNet with ResNet-50 and EfficientNet-L1, and on CIFAR-10.
• Perform ablations with backward-pass only smoothing, forward-pass smoothing, and full SAT to isolate effects.
• Compare SAT against prior art and analyze scalability with network depth, width, and resolution.

Experimental results

Research questions

• RQ1 Does ReLU’s non-smooth gradient degrade adversarial training performance?
• RQ2 Can smooth activations improve adversarial robustness without hurting clean-accuracy or increasing computation?
• RQ3 How does SAT perform with larger networks and different architectures (e.g., EfficientNet) compared to standard adversarial training?
• RQ4 What is the impact of applying smoothing in both forward and backward passes vs only in one pass?

Key findings

• SAT improves ResNet-50 robustness from 33.0% to 42.3% on ImageNet with no accuracy loss or extra cost (0.9% accuracy increase).
• Using smooth activations in both forward and backward passes yields the best robustness gains (up to 3.9% more than ReLU baseline).
• EfficientNet-L1 with SAT achieves 82.2% accuracy and 58.6% robustness on ImageNet, outperforming prior art by 9.5% in accuracy and 11.6% in robustness.
• SAT with smooth activations consistently boosts robustness across tested models while maintaining comparable accuracy.
• SAT enables scaling (depth, width, resolution) to further improve robustness, with compound scaling surpassing standard adversarial training.
• On CIFAR-10, Softplus/GELU/SmoothReLU improve robustness over ReLU; ELU can be unstable unless smoothed (CELU).