[Paper Review] Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error

Stanislav Fort, Andrew Brock+3 more|arXiv (Cornell University)|May 27, 2021

Advanced Neural Network Applications38 references4 citations

TL;DR

This paper proposes increasing augmentation multiplicity—drawing multiple augmented samples per unique image during training—to improve test accuracy in deep ResNets and NFNets. Despite slower training convergence when batch size is fixed, higher augmentation multiplicity consistently boosts generalization, achieving 86.8% top-1 accuracy on ImageNet with NFNet-F5 using only 168,900 parameter updates, outperforming standard training under the same compute budget.

ABSTRACT

In computer vision, it is standard practice to draw a single sample from the data augmentation procedure for each unique image in the mini-batch. However recent work has suggested drawing multiple samples can achieve higher test accuracies. In this work, we provide a detailed empirical evaluation of how the number of augmentation samples per unique image influences model performance on held out data when training deep ResNets. We demonstrate drawing multiple samples per image consistently enhances the test accuracy achieved for both small and large batch training. Crucially, this benefit arises even if different numbers of augmentations per image perform the same number of parameter updates and gradient evaluations (requiring the same total compute). Although prior work has found variance in the gradient estimate arising from subsampling the dataset has an implicit regularization benefit, our experiments suggest variance which arises from the data augmentation process harms generalization. We apply these insights to the highly performant NFNet-F5, achieving 86.8$\%$ top-1 w/o extra data on ImageNet.

Motivation & Objective

• To investigate how the number of data augmentation samples per unique image affects test accuracy in deep learning.
• To disentangle the roles of bias and variance introduced by data augmentation in generalization.
• To evaluate whether the benefits of data augmentation arise from gradient variance or from the biased gradient updates it induces.
• To assess whether augmentation multiplicity improves performance under fixed compute budgets, especially in large-batch training.
• To validate the effectiveness of augmentation multiplicity on state-of-the-art models like NFNet-F5.

Proposed method

• Introduce 'augmentation multiplicity'—drawing multiple augmented versions of each unique image per minibatch—while varying batch size or number of unique images.
• Implement two training schemes: one with growing batch size (fixed number of unique images), and one with fixed batch size (reduced number of unique images per batch) as multiplicity increases.
• Train deep ResNets and NFNets under both schemes, comparing test accuracy and convergence speed.
• Use a fixed total compute budget (e.g., 112,600 or 225,200 parameter updates) to isolate the effect of augmentation multiplicity from training duration.
• Apply SAM regularization to NFNets to evaluate performance under strong optimization regimes.
• Compare results to standard training (augmentation multiplicity = 1) and to prior SOTA models from the literature.

Experimental results

Research questions

• RQ1 Does increasing the number of augmentation samples per unique image improve test accuracy in deep ResNets?
• RQ2 Are the generalization benefits of data augmentation due to the bias in the gradient estimate or the variance introduced by augmentation?
• RQ3 Does the performance gain from higher augmentation multiplicity persist when the total number of parameter updates is held constant?
• RQ4 How does augmentation multiplicity interact with large-batch training and finite learning rates?
• RQ5 Can high augmentation multiplicity reduce the number of training epochs required to achieve SOTA performance?

Key findings

• Increasing augmentation multiplicity from 1 to 16 improves test accuracy on ImageNet by 0.4% for NFNet-F5, achieving 86.8% top-1 accuracy without extra data.
• Even under a fixed compute budget of 112,600 parameter updates, augmentation multiplicity 16 achieves higher test accuracy than standard training with multiplicity 1.
• With a fixed batch size, higher augmentation multiplicity leads to slower training convergence but significantly better generalization, especially in large-batch settings.
• The same model trained with augmentation multiplicity 16 reaches 86.8% top-1 accuracy in 34 epochs (168,900 steps), while the baseline requires more epochs and achieves lower accuracy.
• For NFNet-F3 with SAM, augmentation multiplicity 16 achieves 86.45% top-1 accuracy after 225,200 updates—within 0.05% of an NFNet-F6 baseline but with 4x less compute and fewer parameters.
• The performance gain from augmentation multiplicity is most pronounced when multiple augmentations of the same image are sampled within the same minibatch, rather than across batches.