[Paper Review] Adversarial Examples Are a Natural Consequence of Test Error in Noise

Nic Ford, Justin Gilmer+2 more|arXiv (Cornell University)|Jan 29, 2019

Adversarial Robustness in Machine Learning27 references135 citations

TL;DR

The paper argues that adversarial examples naturally arise from nonzero test error under Gaussian noise, linking adversarial robustness to corruption robustness and recommending joint evaluation with distributional-shift benchmarks like Imagenet-C.

ABSTRACT

Over the last few years, the phenomenon of adversarial examples --- maliciously constructed inputs that fool trained machine learning models --- has captured the attention of the research community, especially when the adversary is restricted to small modifications of a correctly handled input. Less surprisingly, image classifiers also lack human-level performance on randomly corrupted images, such as images with additive Gaussian noise. In this paper we provide both empirical and theoretical evidence that these are two manifestations of the same underlying phenomenon, establishing close connections between the adversarial robustness and corruption robustness research programs. This suggests that improving adversarial robustness should go hand in hand with improving performance in the presence of more general and realistic image corruptions. Based on our results we recommend that future adversarial defenses consider evaluating the robustness of their methods to distributional shift with benchmarks such as Imagenet-C.

Motivation & Objective

• Motivate the connection between adversarial robustness and corruption robustness under noisy/corrupted image distributions.
• Characterize the error set of classifiers under Gaussian noise and relate it to small adversarial perturbations.
• Demonstrate how improvements to adversarial robustness affect corruption robustness and vice versa.
• Provide practical guidance for evaluating defenses using corruption benchmarks in addition to adversarial metrics.

Proposed method

• Define and analyze the error set E of a classifier under clean and corrupted inputs.
• Relate corruption robustness under Gaussian noise to adversarial robustness via distance to E and noise scale parameters.
• Use linear-model intuition to derive a relationship d(x0,E) ≈ -sigma Phi^{-1}(mu) linking distance to the decision boundary with Gaussian noise error rate mu.
• Empirically compare normally trained, Gaussian-augmented, and adversarially trained models on CIFAR-10 and ImageNet.
• Apply Gaussian isoperimetric inequality to bound median distance to the nearest error as a function of noise level and error rate.
• Evaluate corruption robustness on Imagenet-C and compare in-memory vs compressed corruptions to assess brittleness and defense efficacy.

Experimental results

Research questions

• RQ1 Do adversarial examples necessarily exist at small distances from clean inputs given observed error rates in Gaussian noise?
• RQ2 How does corruption robustness under Gaussian perturbations relate to adversarial robustness across different training regimes (natural, Gaussian augmentation, adversarial training)?
• RQ3 Can isoperimetric bounds meaningfully explain the proximity of adversarial examples to clean data under realistic noise distributions?
• RQ4 Do defenses that improve adversarial robustness also improve corruption robustness, and vice versa?
• RQ5 What is the impact of common corruption benchmarks (Imagenet-C) on evaluating adversarial defenses?

Key findings

• Adversarial examples appear at distances predicted by a half-space error model under Gaussian noise, implying no need for exotic decision boundaries to explain small perturbations.
• Improving adversarial robustness (via adversarial training) also increases robustness to Gaussian noise, while Gaussian augmentation mainly increases robustness to noise and to some extent small perturbations.
• Gaussian isoperimetric inequality shows that for a given nonzero Gaussian-noise error rate, the median distance to the nearest error is bounded by -sigma Phi^{-1}(mu), with equality for half-spaces, aligning with empirical observations.
• Models with higher error rates in Gaussian noise have closer nearby errors, and the most robust models tend toward half-space-like error sets, consistent with linear intuition.
• Adversarial training and Gaussian augmentation both improve corruption robustness and can alter robustness profiles across different corruption types (e.g., blur, noise, fog, contrast).
• Corruption robustness benchmarks (Imagenet-C) reveal that defenses may perform differently in compressed vs in-memory corruptions, highlighting brittleness under distributional shift and the value of cross-evaluation.