[Paper Review] Does Data Augmentation Benefit from Split BatchNorms

Amil Merchant, Barret Zoph+1 more|arXiv (Cornell University)|Oct 15, 2020

Stochastic Gradient Optimization Techniques41 references4 citations

TL;DR

This paper investigates whether split BatchNorm—using separate statistics for clean and augmented images—improves model performance in data augmentation. It finds that using weak augmentations (not clean images) to define the main BatchNorm significantly boosts accuracy on CIFAR-10, CIFAR-100, and ImageNet, while maintaining robustness through prediction averaging, revealing a trade-off between accuracy and corruption robustness.

ABSTRACT

Data augmentation has emerged as a powerful technique for improving the performance of deep neural networks and led to state-of-the-art results in computer vision. However, state-of-the-art data augmentation strongly distorts training images, leading to a disparity between examples seen during training and inference. In this work, we explore a recently proposed training paradigm in order to correct for this disparity: using an auxiliary BatchNorm for the potentially out-of-distribution, strongly augmented images. Our experiments then focus on how to define the BatchNorm parameters that are used at evaluation. To eliminate the train-test disparity, we experiment with using the batch statistics defined by clean training images only, yet surprisingly find that this does not yield improvements in model performance. Instead, we investigate using BatchNorm parameters defined by weak augmentations and find that this method significantly improves the performance of common image classification benchmarks such as CIFAR-10, CIFAR-100, and ImageNet. We then explore a fundamental trade-off between accuracy and robustness coming from using different BatchNorm parameters, providing greater insight into the benefits of data augmentation on model performance.

Motivation & Objective

• To address the train-test distribution disparity caused by aggressive data augmentation in deep learning.
• To investigate whether split BatchNorm—using separate statistics for clean and augmented images—can mitigate this disparity.
• To determine the optimal definition of BatchNorm parameters at inference time for improved model performance.
• To analyze the trade-off between accuracy and robustness when using different BatchNorm parameter definitions.

Proposed method

• Adopt a split BatchNorm training paradigm where an auxiliary BatchNorm is used for strongly augmented, out-of-distribution images.
• Define the main BatchNorm using statistics from weakly augmented images rather than clean images.
• Use a weighted average of predictions from the main and auxiliary BatchNorm layers at inference to balance accuracy and robustness.
• Evaluate model performance on clean accuracy, corruption robustness (Common Corruptions), and Fourier sensitivity to noise.
• Apply low-pass filtering in the Fourier domain to analyze frequency content usage and explain performance gains.
• Conduct ablation studies to isolate the contribution of weak augmentations to performance improvements.

Experimental results

Research questions

• RQ1 Does using clean (un-augmented) images to define the main BatchNorm improve model performance in a split BatchNorm setup?
• RQ2 Can weak augmentations be used effectively to define the main BatchNorm, leading to better performance than clean images?
• RQ3 What is the trade-off between model accuracy and robustness when using different BatchNorm parameter definitions?
• RQ4 How does the frequency content of augmented images relate to performance improvements in the split BatchNorm setup?

Key findings

• Using clean images to define the main BatchNorm in a split BatchNorm setup does not improve performance and may harm it due to overfitting.
• Defining the main BatchNorm using weak augmentations leads to significant improvements in clean accuracy on CIFAR-10, CIFAR-100, and ImageNet benchmarks.
• The auxiliary BatchNorm (trained on strongly augmented images) shows higher sensitivity to high-frequency noise, indicating reduced robustness.
• Averaging predictions from the main and auxiliary BatchNorm layers preserves most of the accuracy gain from strong augmentation while restoring robustness.
• Weak augmentations allow models to better utilize higher-frequency information in training data, as shown by Fourier domain filtering experiments.
• The model trained with weak augmentations to define the main BatchNorm effectively interpolates between the frequency usage of no augmentation and strong augmentation (e.g., RandAugment).