[Paper Review] On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness

Eric Mintun, Alexander Kirillov+1 more|arXiv (Cornell University)|Feb 22, 2021

Domain Adaptation and Few-Shot LearningComputer Science31 citations

TL;DR

The paper introduces a perceptual similarity framework between data augmentations and test-time corruptions using a Minimal Sample Distance (MSD) measure, showing that augmentation-corruption similarity predicts corruption error and that generalization to dissimilar corruptions is limited.

ABSTRACT

Invariance to a broad array of image corruptions, such as warping, noise, or color shifts, is an important aspect of building robust models in computer vision. Recently, several new data augmentations have been proposed that significantly improve performance on ImageNet-C, a benchmark of such corruptions. However, there is still a lack of basic understanding on the relationship between data augmentations and test-time corruptions. To this end, we develop a feature space for image transforms, and then use a new measure in this space between augmentations and corruptions called the Minimal Sample Distance to demonstrate a strong correlation between similarity and performance. We then investigate recent data augmentations and observe a significant degradation in corruption robustness when the test-time corruptions are sampled to be perceptually dissimilar from ImageNet-C in this feature space. Our results suggest that test error can be improved by training on perceptually similar augmentations, and data augmentations may not generalize well beyond the existing benchmark. We hope our results and tools will allow for more robust progress towards improving robustness to image corruptions. We provide code at https://github.com/facebookresearch/augmentation-corruption.

Motivation & Objective

• Understand how data augmentations influence robustness to image corruptions.
• Develop a perceptual feature space for transforms to quantify augmentation-corruption similarity.
• Define and validate Minimal Sample Distance (MSD) as a distance measure relevant to corruption robustness.
• Investigate generalization of augmentation schemes to dissimilar corruptions using new benchmarks.
• Provide practical guidance for evaluating and designing robust augmentation strategies.

Proposed method

• Create a perceptual transform feature space using the last hidden layer of a pretrained classifier to encode transforms.
• Define Minimal Sample Distance (MSD) as the distance between the closest augmentation sample and the mean feature of a corruption distribution.
• Compute MSD between a wide set of augmentation schemes (the augmentation powerset) and CIFAR-10-C/ImageNet-C corruptions and correlate with corruption error.
• Compare MSD with MMD to argue MSD better predicts robustness to unseen corruptions.
• Evaluate existing augmentation methods (AutoAugment, Patch Gaussian, AugMix, etc.) against CIFAR-10-C and ImageNet-C and introduce ImageNet-ar{C} and CIFAR-10-ar{C} as dissimilar benchmarks.
• Analyze degradation of augmentation performance on dissimilar corruptions to assess generalization.

Experimental results

Research questions

• RQ1 How does perceptual similarity between train-time augmentations and test-time corruptions relate to corruption error?
• RQ2 Can MSD predict corruption robustness better than traditional distribution distance measures (e.g., MMD)?
• RQ3 Do augmentation schemes that are perceptually similar to common corruptions generalize to dissimilar corruptions?

Key findings

• MSD correlates strongly with corruption error across CIFAR-10-C corruptions, with many corruptions showing high Spearman correlations (e.g., 12 of 15 > 0.6).
• MMD does not reliably predict corruption error, especially for broad augmentations.
• Perceptually similar augmentations can improve robustness to similar corruptions even if the exact corruption is not present in training.
• New corruptions (ImageNet-ar{C} and CIFAR-10-ar{C}), selected for dissimilarity in the transform feature space, reveal that several augmentation schemes degrade on dissimilar corruptions, highlighting limited generalization.
• Broad augmentation methods (AutoAugment, AugMix, Stylized-ImageNet, DeepAugment) tend to generalize better to dissimilar corruptions than narrower schemes.
• Even robust models not primarily augmentation-based do not show the same dissimilarity degradation pattern, suggesting the observed effects are tied to augmentation-based robustness.