[Paper Review] An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods

Sanghyuk Chun, Seong Joon Oh+4 more|arXiv (Cornell University)|Mar 9, 2020

Adversarial Robustness in Machine Learning42 references35 citations

TL;DR

The paper empirically evaluates state-of-the-art regularization techniques (e.g., Mixup, CutMix, Cutout, ShakeDrop, label smoothing) for robustness to adversarial and natural perturbations and for uncertainty estimation, showing they can outperform specialized robustness methods as strong, cheap baselines.

ABSTRACT

Despite apparent human-level performances of deep neural networks (DNN), they behave fundamentally differently from humans. They easily change predictions when small corruptions such as blur and noise are applied on the input (lack of robustness), and they often produce confident predictions on out-of-distribution samples (improper uncertainty measure). While a number of researches have aimed to address those issues, proposed solutions are typically expensive and complicated (e.g. Bayesian inference and adversarial training). Meanwhile, many simple and cheap regularization methods have been developed to enhance the generalization of classifiers. Such regularization methods have largely been overlooked as baselines for addressing the robustness and uncertainty issues, as they are not specifically designed for that. In this paper, we provide extensive empirical evaluations on the robustness and uncertainty estimates of image classifiers (CIFAR-100 and ImageNet) trained with state-of-the-art regularization methods. Furthermore, experimental results show that certain regularization methods can serve as strong baseline methods for robustness and uncertainty estimation of DNNs.

Motivation & Objective

• Assess how common regularization techniques affect robustness to adversarial perturbations and natural corruptions.
• Evaluate calibration and OOD detection performance of regularized models.
• Compare regularization-based baselines against methods specifically designed for robustness and uncertainty.
• Identify combinations of regularizations that yield strong overall robustness with minimal cost.
• Highlight implications for using regularization as a baseline in robustness research.

Proposed method

• Train image classifiers on CIFAR-100 and ImageNet with PyramidNet-200 and ResNet-50 baselines.
• Apply regularization techniques: Cutout, Mixup, CutMix, ShakeDrop, Label Smoothing and their combinations.
• Evaluate robustness to FGSM adversarial attack, CIFAR-100-C/ImageNet-C corruptions, and occlusions.
• Assess uncertainty via expected calibration error and OOD detection error.
• Compare against adversarial training and adversarial logit pairing (ALP).
• Use standard SGD with momentum (batch=64, 300 epochs) and data augmentations; report results with label smoothing.
• Utilize OOD augmentations (SVHN, GAN) and CIFAR-C/ImageNet-C benchmarks for uncertainty tests.

Experimental results

Research questions

• RQ1 Do simple, well-known regularization techniques provide robust performance across adversarial and non-adversarial perturbations?
• RQ2 How do regularization methods affect calibration and OOD detection compared to specialized robustness methods?
• RQ3 Can combinations of regularization methods outperform targeted robustness approaches on common benchmarks?
• RQ4 What is the relative impact of label smoothing on robustness and uncertainty estimates in regularized models?
• RQ5 Are regularized models viable strong baselines for robustness and uncertainty research?

Key findings

• Regularized models with combinations like Cutout + ShakeDrop + label smoothing, Mixup + ShakeDrop + label smoothing, and CutMix + ShakeDrop + label smoothing improve robustness and uncertainty outcomes across metrics.
• Label smoothing alone often does not improve classification accuracy but improves adversarial robustness, calibration error, and OOD detection in several setups.
• Some regularization combinations surpass specialized methods (e.g., ALP, OOD augmentations) on multiple metrics, particularly in occlusion and uncertainty measures.
• Mixup and CutMix generally improve robustness to common corruptions; Cutout excels in occlusion robustness but may hurt other perturbations.
• On ImageNet, regularized models achieve competitive robustness, with Mixup + label smoothing attaining a strong ImageNet-C mCE (69.14%) comparable to stylized-ImageNet baselines.
• Overall, well-regularized models provide stronger or comparable robustness and uncertainty performance at negligible additional cost, suggesting they should be strong baselines in future work.