[Paper Review] Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations

Dan Hendrycks, Thomas G. Dietterich|arXiv (Cornell University)|Jul 4, 2018

Adversarial Robustness in Machine LearningComputer Science55 references112 citations

TL;DR

This paper introduces ImageNet-C for corruption robustness and Icons-50 for surface-variation robustness, benchmarking modern nets and proposing methods to improve robustness beyond clean accuracy alone.

ABSTRACT

In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Unlike recent robustness research, this benchmark evaluates performance on commonplace corruptions not worst-case adversarial corruptions. We find that there are negligible changes in relative corruption robustness from AlexNet to ResNet classifiers, and we discover ways to enhance corruption robustness. Then we propose a new dataset called Icons-50 which opens research on a new kind of robustness, surface variation robustness. With this dataset we evaluate the frailty of classifiers on new styles of known objects and unexpected instances of known classes. We also demonstrate two methods that improve surface variation robustness. Together our benchmarks may aid future work toward networks that learn fundamental class structure and also robustly generalize.

Motivation & Objective

• Define rigorous benchmarks for image classifier robustness to common corruptions and surface variations.
• Create ImageNet-C to standardize corruption robustness across 75 perturbations (15 types x 5 severities).
• Create Icons-50 to study robustness to style and subtype surface variations.
• Evaluate current architectures and robustness-enhancing methods on these benchmarks.
• Propose techniques that improve robustness without sacrificing accuracy.

Proposed method

• Introduce ImageNet-C with 15 corruption types, five severity levels each, applied to ImageNet validation images.
• Define Corruption Error and Mean Corruption Error (mCE), and Relative Corruption Error and Relative mCE for cross-type comparability.
• Evaluate a broad set of architectures (e.g., AlexNet, VGG variants, ResNet, DenseNet, ResNeXt, Multigrid, MSDNet) on ImageNet-C.
• Experiment with robustness-enhancing techniques (stability training, denoising, histogram equalization) and analyze their impact.
• Investigate multiscale and feature-aggregation networks (Multigrid, MSDNet, DenseNet, ResNeXt) on corruption robustness.
• Develop the Icons-50 dataset to study surface variation robustness (style and subtype robustness) and propose evaluation protocols.
• Explore improvements via Shake-Shake regularization and multiscale networks to enhance surface variation robustness.

Experimental results

Research questions

• RQ1 How do common image corruptions affect top-1 accuracy across standard architectures?
• RQ2 Do architectural improvements improve corruption robustness, or is robustness mostly coupled to clean accuracy gains?
• RQ3 Can preprocessing or training-time techniques meaningfully improve robustness to the 75 ImageNet-C corruptions?
• RQ4 What is the effect of multiscale and feature-aggregation architectures on robustness to noise and surface variations?
• RQ5 How robust are networks to surface variation such as unseen styles and subtypes, as tested with Icons-50?

Key findings

• ImageNet-C reveals modest improvements in corruption robustness across architectures; relative robustness largely tracks clean accuracy gains.
• Mean Corruption Error (mCE) increases with architectural progress, but Relative mCE often remains near AlexNet levels, indicating limited intrinsic robustness gains.
• Histogram equalization and multiscale/large architectures provide notable robustness gains on ImageNet-C without sacrificing clean accuracy.
• Stability training and input denoising did not improve robustness on ImageNet-C in their experiments, and denoising sometimes worsened performance.
• Shake-Shake regularization and multiscale networks (Multigrid, MSDNet) improve both style and subtype robustness on Icons-50; larger feature aggregation generally enhances robustness beyond pure depth growth.
• Icons-50 shows current models struggle with style robustness (e.g., unseen icon sources) and subtype robustness (held-out subtypes), indicating surface variation robustness is a distinct challenge from corruption robustness.