[论文解读] Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Dan Hendrycks, Thomas G. Dietterich|arXiv (Cornell University)|Mar 28, 2019

Adversarial Robustness in Machine Learning被引用 785

一句话总结

简要: Introduces ImageNet-C and ImageNet-P benchmarks to evaluate image classifier robustness to common corruptions and perturbations, comparing architectures and suggesting robustness improvements beyond clean accuracy.

ABSTRACT

In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize.

研究动机与目标

• Motivate the need for robustness benchmarks beyond adversarial examples.
• Define corruption and perturbation robustness in image classification.
• Create and release ImageNet-C (corruptions) and ImageNet-P (perturbations) datasets.
• Propose metrics to quantify corruption and perturbation robustness and baseline results across architectures.
• Demonstrate methods that improve robustness and reveal interactions with adversarial defenses.

提出的方法

• Define corruption robustness as the average performance across 75 corruptions at five severity levels on ImageNet validation data.
• Define perturbation robustness via perturbation sequences (ImageNet-P) and metrics like Flip Rate and Top-5 Distance.
• Introduce ImageNet-C with 15 corruption types across four categories (noise, blur, weather, digital) and five severity levels.
• Introduce ImageNet-P with temporally sequenced perturbations across selected perturbation types and evaluation metrics.
• Evaluate multiple architectures (e.g., AlexNet, SqueezeNet, VGG, ResNet, DenseNet, ResNeXt) to assess robustness trends.
• Explore robustness enhancements (CLAHE, multiscale networks, larger feature aggregation, stylization augmentation, and ALP) and report interactions with adversarial defenses.

实验结果

研究问题

• RQ1 常见损坏和扰动如何影响不同架构的图像分类器性能？
• RQ2 干净准确度的提升是否会转化为对损坏和扰动的鲁棒性提升？
• RQ3 是否可以通过特定的架构或预处理变更在不损失准确度的情况下提高损坏和扰动鲁棒性？
• RQ4 对抗性防御与对常见扰动鲁棒性之间有什么关系？
• RQ5 哪些基线指标最能准确捕捉鲁棒性并实现跨模型公平比较？

主要发现

• 从 AlexNet 到 ResNet 的架构进展对损坏鲁棒性的提升有限（mCE 改进适中且常与干净准确度相关）。
• 扰动鲁棒性研究不足，甚至对强模型也会退化；顶级预测在常见扰动下可能不稳定。
• 多尺度和特征聚合架构（DenseNets、ResNeXts、Multigrid）在损坏鲁棒性方面相对于原始 ResNet 有显著提升。
• 更大、更单元化的模型具有更高冗余，能够在不纯粹追求准确度的情况下改善对噪声和失真鲁棒性。
• CLAHE 预处理对损坏鲁棒性有温和提升；基于风格化的增强和 ALP 防御也可增强对常见扰动的鲁棒性。
• 鲁棒性提升可来自架构变更和定向预处理/增强，某些对抗防御可提供跨鲁棒性的好处。