[Paper Review] Improving robustness against common corruptions by covariate shift adaptation

Steffen Schneider, Evgenia Rusak+4 more|arXiv (Cornell University)|Jun 30, 2020

Domain Adaptation and Few-Shot LearningComputer Science61 references161 citations

TL;DR

The paper shows that adapting batch normalization statistics to unlabeled corrupted images greatly improves robustness to common corruptions across many models, and proposes evaluation variants and a simple Wasserstein-distance-based analysis.

ABSTRACT

Today's state-of-the-art machine vision models are vulnerable to image corruptions like blurring or compression artefacts, limiting their performance in many real-world applications. We here argue that popular benchmarks to measure model robustness against common corruptions (like ImageNet-C) underestimate model robustness in many (but not all) application scenarios. The key insight is that in many scenarios, multiple unlabeled examples of the corruptions are available and can be used for unsupervised online adaptation. Replacing the activation statistics estimated by batch normalization on the training set with the statistics of the corrupted images consistently improves the robustness across 25 different popular computer vision models. Using the corrected statistics, ResNet-50 reaches 62.2% mCE on ImageNet-C compared to 76.7% without adaptation. With the more robust DeepAugment+AugMix model, we improve the state of the art achieved by a ResNet50 model up to date from 53.6% mCE to 45.4% mCE. Even adapting to a single sample improves robustness for the ResNet-50 and AugMix models, and 32 samples are sufficient to improve the current state of the art for a ResNet-50 architecture. We argue that results with adapted statistics should be included whenever reporting scores in corruption benchmarks and other out-of-distribution generalization settings.

Motivation & Objective

• Motivate that robustness benchmarks may underestimate real-world performance when unlabeled corrupted data are available.
• Propose unsupervised adaptation of batch normalization statistics to reduce covariate shift from clean to corrupted images.
• demonstrate robustness gains across a wide range of architectures and datasets.
• Provide practical guidance on when and how many samples are needed for adaptation.
• Introduce metrics and theoretical insights to relate covariate shift to degradation via Wasserstein distance.

Proposed method

• Adapt BN statistics by computing target statistics from unlabeled corrupted samples and combining them with training statistics using a pseudo-sample size parameter N.
• Evaluate robustness via mean corruption error (mCE) under ad hoc (n=1), partial (n=8), and full (n=50,000) adaptation scenarios.
• Test on ImageNet-C across 25 architectures and compare to state-of-the-art robustness methods.
• Analyze the relationship between covariate shift and performance using Wasserstein distance between source and target statistics.
• Explore cases where BN adaptation is not beneficial (e.g., IN-A, ON) and compare with alternatives like GN/Fixup.
• Provide a simple bound/model linking the adaptation parameters with expected covariance-shift-related degradation.

Experimental results

Research questions

• RQ1 Does adapting batch normalization statistics on unlabeled corrupted data improve robustness to common corruptions across diverse architectures?
• RQ2 How does the amount of adaptation data (pseudo-sample size N and number of samples n) affect robustness gains?
• RQ3 Is the improvement consistent across different corruption types and datasets beyond ImageNet-C?
• RQ4 Can the covariate shift captured by BN statistics be characterized and predicted by a Wasserstein distance between source and target distributions?
• RQ5 In which scenarios does BN adaptation fail or underperform compared to alternative normalization schemes?

Key findings

• Adapting BN statistics yields substantial robustness gains, often around 10 percentage points in mCE across 25 architectures.
• For Vanilla ResNet-50, adaptation reduces mCE from 76.7% to 62.2% (full adaptation) and to 65.0% (partial adaptation).
• Even adapting to a single sample can improve performance (e.g., mCE improves from 76.7% to 71.4% with N≈0 and n=1).
• State-of-the-art robustness on IN-C for ResNet-50 can be surpassed by adapting BN, e.g., DeepAugment+AugMix improves to 45.4% mCE (from 53.6% non-adapted).
• Across 25 model families, BN adaptation consistently improves mCE, typically by around 10 points; larger pretraining (e.g., IG-3.5B) can reduce or remove the need for adaptation.
• The Wasserstein distance between source and target BN statistics correlates with top-1 error both before and after adaptation, enabling unsupervised performance estimation.
• Adaptation effectiveness varies by dataset; BN adaptation is less effective for IN-A and ObjectNet, where learned features shift differently or non-BN-based robustness is favored.