[Paper Review] On the effectiveness of adversarial training against common corruptions

Klim Kireev, Maksym Andriushchenko+1 more|arXiv (Cornell University)|Mar 3, 2021

Adversarial Robustness in Machine Learning54 references17 citations

TL;DR

This paper demonstrates that $ε$-adversarial training with a small perturbation radius ($\varepsilon = 1/255$) significantly improves robustness to common image corruptions on CIFAR-10 and ImageNet-100, outperforming standard baselines like Gaussian augmentation. It introduces a novel, efficient relaxation of adversarial training using learned perceptual similarity (LPIPS) as a distance metric, which achieves state-of-the-art performance when combined with data augmentation methods such as AugMix and DeepAugment.

ABSTRACT

The literature on robustness towards common corruptions shows no consensus on whether adversarial training can improve the performance in this setting. First, we show that, when used with an appropriately selected perturbation radius, $\ell_p$ adversarial training can serve as a strong baseline against common corruptions improving both accuracy and calibration. Then we explain why adversarial training performs better than data augmentation with simple Gaussian noise which has been observed to be a meaningful baseline on common corruptions. Related to this, we identify the $σ$-overfitting phenomenon when Gaussian augmentation overfits to a particular standard deviation used for training which has a significant detrimental effect on common corruption accuracy. We discuss how to alleviate this problem and then how to further enhance $\ell_p$ adversarial training by introducing an efficient relaxation of adversarial training with learned perceptual image patch similarity as the distance metric. Through experiments on CIFAR-10 and ImageNet-100, we show that our approach does not only improve the $\ell_p$ adversarial training baseline but also has cumulative gains with data augmentation methods such as AugMix, DeepAugment, ANT, and SIN, leading to state-of-the-art performance on common corruptions. The code of our experiments is publicly available at https://github.com/tml-epfl/adv-training-corruptions.

Motivation & Objective

• . The paper aims to resolve the ambiguity in the literature about whether adversarial training improves robustness to common image corruptions.
• It investigates why adversarial training outperforms simple Gaussian data augmentation, identifying a key issue: $σ$-overfitting in Gaussian augmentation.
• The objective is to develop a more effective and efficient adversarial training method that generalizes better to common corruptions.
• It seeks to combine the new method with existing data augmentation techniques to achieve state-of-the-art performance on standard benchmarks.
• The study aims to improve both accuracy and calibration under distribution shift, particularly on corrupted images.

Proposed method

• . The authors use $ε$-adversarial training with carefully selected perturbation radii, finding that $\varepsilon = 1/255$ yields superior performance compared to the standard $\varepsilon = 8/255$.
• They introduce a novel relaxation of adversarial training that uses learned perceptual image patch similarity (LPIPS) as the distance metric, computed via layerwise adversarial perturbations.
• The method is designed to be computationally efficient while maintaining strong robustness, outperforming previous approaches like Fast PAT and standard $ε$-adversarial training.
• The approach is combined with established data augmentation techniques such as AugMix, DeepAugment, ANT, and SIN to achieve cumulative gains.
• The authors perform extensive ablation studies across multiple architectures (ResNet-18, WRN-28-10) and datasets (CIFAR-10-C, ImageNet-100-C) to validate generalization.
• Calibration is evaluated using expected calibration error (ECE), with temperature scaling applied to improve reliability.

Experimental results

Research questions

• RQ1 . Does $ε$-adversarial training with a small perturbation radius improve robustness to common image corruptions compared to standard settings?
• RQ2 Why does adversarial training outperform Gaussian data augmentation, despite both being data-augmentation-like methods?
• RQ3 What causes the performance degradation in Gaussian augmentation, and can it be mitigated?
• RQ4 Can a learned perceptual similarity metric improve adversarial training efficiency and robustness?
• RQ5 Does combining the proposed method with existing data augmentation techniques lead to cumulative improvements on standard corruption benchmarks?

Key findings

• . $ε$-adversarial training with $\varepsilon = 1/255$ achieves 84.8% accuracy on CIFAR-10-C, significantly outperforming the standard $\varepsilon = 8/255$ setting.
• The proposed LPIPS-based relaxation (RLAT) achieves 85.9% accuracy on CIFAR-10-C with WRN-28-10, outperforming other adversarial training methods.
• On CIFAR-10-C, combining RLAT with AugMix reduces ECE to 1.3%, the lowest among all evaluated methods.
• On ImageNet-100-C, the combination of RLAT with ANT 3x3 reduces ECE to 2.8%, compared to 4.5% for ANT alone.
• The method shows strong generalization, as it improves performance across multiple architectures and datasets.
• The study confirms that adversarial training with small $\varepsilon$ improves calibration on corrupted data (ECE reduced from 16.6% to 9.9%) while only slightly degrading it on clean data (from 2.9% to 4.0% for RLAT).