[Paper Review] Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data

Zhuoxun He, Lingxi Xie+4 more|arXiv (Cornell University)|Sep 19, 2019

Advanced Neural Network Applications47 references35 citations

TL;DR

The paper analyzes data augmentation as regularization, derives an upper-bound on expected risk, and proposes refining augmentation by ending training with less or no augmentation to close the distribution gap, yielding consistent gains on classification and improved transfer to detection.

ABSTRACT

Data augmentation has been widely applied as an effective methodology to improve generalization in particular when training deep neural networks. Recently, researchers proposed a few intensive data augmentation techniques, which indeed improved accuracy, yet we notice that these methods augment data have also caused a considerable gap between clean and augmented data. In this paper, we revisit this problem from an analytical perspective, for which we estimate the upper-bound of expected risk using two terms, namely, empirical risk and generalization error, respectively. We develop an understanding of data augmentation as regularization, which highlights the major features. As a result, data augmentation significantly reduces the generalization error, but meanwhile leads to a slightly higher empirical risk. On the assumption that data augmentation helps models converge to a better region, the model can benefit from a lower empirical risk achieved by a simple method, i.e., using less-augmented data to refine the model trained on fully-augmented data. Our approach achieves consistent accuracy gain on a few standard image classification benchmarks, and the gain transfers to object detection.

Motivation & Objective

• Motivate why intensive data augmentation creates a distribution gap between clean and augmented data
• Reformulate the upper bound of expected risk to separate empirical risk and generalization error under augmentation
• Explain augmentation as a regularization effect on minor features while preserving major features
• Propose a refined training strategy that reduces augmentation towards the end of training
• Demonstrate improvements across standard benchmarks and transfer to detection tasks

Proposed method

• Formulate expected risk with data augmentation and distinguish between P and P_aug
• Analyze convergence of empirical risk under intensive augmentation using a feature-space decomposition into major and minor features
• Use Taylor expansion to justify how intensive augmentation regularizes minor features and constrains corresponding weights
• Propose refined data augmentation: continue standard augmentation during refinement but reduce or stop intensive augmentation at the end of training
• Empirically validate on CIFAR-10/100, Tiny-ImageNet, ImageNet, and Pascal VOC with Faster R-CNN to test classification and detection transfer

Experimental results

Research questions

• RQ1 How does data augmentation influence the upper bound of expected risk via empirical risk and generalization error?
• RQ2 Can intensive augmentation be reconciled with convergence of empirical risk when a large distribution gap exists?
• RQ3 Does a training schedule that reduces augmentation at the end improve final generalization and transfer to detection?
• RQ4 What are the practical gains of refined augmentation across standard benchmarks and detection tasks?

Key findings

• Intensive augmentation reduces generalization error but can increase empirical risk on augmented data.
• Refined augmentation, i.e., ending with less augmentation, yields further reductions in empirical risk and improved test performance.
• Augmentation helps models converge to better regions of parameter space, and refinement can preserve or enhance gains across CIFAR, Tiny-ImageNet, and ImageNet.
• Refinement improves transfer to object detection, suggesting end-of-training augmentation helps recover location-sensitive features.
• The benefits persist across various augmentation types including Mixup, Manifold Mixup, CutMix, and AutoAugment, with refinement mitigating overfitting to augmented data.