[Paper Review] Natural Adversarial Examples

Dan Hendrycks, Kevin Zhao+3 more|arXiv (Cornell University)|Jul 16, 2019

Adversarial Robustness in Machine Learning77 references193 citations

TL;DR

The paper introduces two real-world, adversarially filtered datasets, ImageNet-A and ImageNet-O, showing large robustness gaps of ImageNet models and that improvements require architectural changes rather than simple data augmentation. The work demonstrates transfer of these hard examples across models including transformers, highlighting shared weaknesses.

ABSTRACT

We introduce two challenging datasets that reliably cause machine learning model performance to substantially degrade. The datasets are collected with a simple adversarial filtration technique to create datasets with limited spurious cues. Our datasets' real-world, unmodified examples transfer to various unseen models reliably, demonstrating that computer vision models have shared weaknesses. The first dataset is called ImageNet-A and is like the ImageNet test set, but it is far more challenging for existing models. We also curate an adversarial out-of-distribution detection dataset called ImageNet-O, which is the first out-of-distribution detection dataset created for ImageNet models. On ImageNet-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%, and its out-of-distribution detection performance on ImageNet-O is near random chance levels. We find that existing data augmentation techniques hardly boost performance, and using other public training datasets provides improvements that are limited. However, we find that improvements to computer vision architectures provide a promising path towards robust models.

Motivation & Objective

• Motivate robust evaluation under distribution shift using natural, real-world adversarial examples.
• Create challenging test sets (ImageNet-A and ImageNet-O) that reduce reliance on spurious cues.
• Assess whether data augmentation, extra data, or architectural changes improve robustness to these shifts.
• Show that improvements transfer across model families, including transformers, and identify promising directions.

Proposed method

• Adversarial filtration to curate ImageNet-A by removing images that fixed ResNet-50 correctly and selecting low-confidence misclassifications.
• Adversarial filtration for ImageNet-O by filtering ImageNet-22K to images that ResNet-50 misclassifies with high confidence as in-distribution.
• Manual curation to ensure single-label, high-quality images for both datasets.
• Evaluation of robustness using top-1 accuracy on ImageNet-A and out-of-distribution detection via AUPR on ImageNet-O.
• Baseline comparisons with standard data augmentations, pretraining, and architectural changes.
• Cross-architecture evaluation including ResNet family, ResNeXt, SE blocks, Res2Net, and vision transformers (DeiT).

Experimental results

Research questions

• RQ1 How do naturally occurring, adversarially filtered images impact ImageNet classifiers across architectures?
• RQ2 Do standard data augmentations or additional labeled data meaningfully improve performance on ImageNet-A and ImageNet-O?
• RQ3 Can architectural changes (e.g., wider networks, Res2Net, self-attention) meaningfully enhance robustness and OOD detection on these datasets?
• RQ4 Do adversarially filtered examples transfer to unseen models, including non-convolutional architectures like vision transformers?
• RQ5 What failure modes do ImageNet-A examples reveal about current CNNs and their reliance on non-semantic cues?

Key findings

• ImageNet-A causes approximately 2% accuracy for DenseNet-121, a ~90% drop compared to typical ImageNet-1K performance.
• ImageNet-O purposes out-of-distribution detection, with OOD anomaly scores generally at random-chance levels on baseline detectors.
• Data augmentation alone yields only modest gains (e.g., various methods provide single-digit increases; best reported ~7-8% in some contexts), and training with an order of magnitude more data yields diminishing returns.
• Pretraining on ImageNet-21K followed by fine-tuning on ImageNet-1K raises ImageNet-A accuracy to 11.41% and ImageNet-O AUPR to 21.86% (vs ~2.17% ImageNet-A baseline).
• Architectural changes produce larger robustness gains than data augmentation or more data (e.g., ResNet-50 to ResNet-152: ImageNet-A from 2.17% to 6.1%; ResNeXt-50(32×4d) to higher capacity variants; Res2Net and self-attention variants substantially improve both ImageNet-A accuracy and ImageNet-O AUPR; Vision Transformers (DeiT) also show transfer to these tasks).
• Self-attention and multi-scale/backbone innovations (e.g., Res2Net v1b, SE blocks) yield notable improvements in ImageNet-A and ImageNet-O, with larger gains on bigger models (e.g., DeiT-base reaching 28.2% on ImageNet-A and 24.8% AUPR for ImageNet).