[Paper Review] Sample Efficiency of Data Augmentation Consistency Regularization

Shuo Yang, Yijun Dong+4 more|arXiv (Cornell University)|Feb 24, 2022

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper provides a theoretical analysis demonstrating that Data Augmentation Consistency (DAC) regularization is intrinsically more sample-efficient than Data Augmentation Empirical Risk Minimization (DA-ERM) in both linear and non-linear models. It shows DAC reduces generalization error by enforcing stronger invariance and better tolerating label-incorrect augmentations, validated empirically on CIFAR-100 with WideResNet.

ABSTRACT

Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to use augmented data. In this paper, we take a step in this direction - we first present a simple and novel analysis for linear regression with label invariant augmentations, demonstrating that data augmentation consistency (DAC) is intrinsically more efficient than empirical risk minimization on augmented data (DA-ERM). The analysis is then extended to misspecified augmentations (i.e., augmentations that change the labels), which again demonstrates the merit of DAC over DA-ERM. Further, we extend our analysis to non-linear models (e.g., neural networks) and present generalization bounds. Finally, we perform experiments that make a clean and apples-to-apples comparison (i.e., with no extra modeling or data tweaks) between DAC and DA-ERM using CIFAR-100 and WideResNet; these together demonstrate the superior efficacy of DAC.

Motivation & Objective

• To theoretically compare the sample efficiency of Data Augmentation Consistency (DAC) versus Data Augmentation Empirical Risk Minimization (DA-ERM), which are commonly used in training deep models.
• To investigate whether DAC offers intrinsic algorithmic advantages over DA-ERM even when no unlabeled data is used.
• To extend theoretical analysis from linear models to non-linear models, including two-layer neural networks and DNN-based classifiers with expansion-based augmentations.
• To provide generalization bounds under consistency regularization for non-linear settings, linking theory to practical design choices.
• To perform a clean, apples-to-apples empirical comparison of DAC and DA-ERM on CIFAR-100 using WideResNet, without additional data or model modifications.

Proposed method

• Proposes a novel theoretical framework for linear regression with label-invariant augmentations to compare DAC and DA-ERM, showing DAC yields strictly smaller generalization error.
• Extends analysis to misspecified augmentations (where augmentations alter labels), demonstrating DAC’s superior robustness and smaller approximation error.
• Derives generalization bounds for DAC in non-linear models, including two-layer ReLU networks and DNN-based classifiers with expansion-based augmentations.
• Uses Gaussian width and Lipschitz continuity to bound the complexity of augmented data manifolds, linking model complexity to generalization performance.
• Employs RandAugment with controlled strength (n ∈ {1,2,5,10}) to generate augmented samples in experiments, ensuring consistent augmentation policies.
• Conducts ablation studies on labeled data sizes (1k, 10k, 20k) and compares DAC and DA-ERM under identical training protocols on CIFAR-100 with WideResNet.

Experimental results

Research questions

• RQ1 Is Data Augmentation Consistency (DAC) intrinsically more sample-efficient than DA-ERM in the absence of unlabeled data?
• RQ2 How does DAC compare to DA-ERM in terms of estimation and approximation error when augmentations are misspecified (i.e., change labels)?
• RQ3 Can theoretical generalization bounds be derived for consistency regularization in non-linear models like two-layer neural networks?
• RQ4 How do hyperparameter choices, such as augmentation strength, affect the performance gap between DAC and DA-ERM?
• RQ5 To what extent does DAC outperform DA-ERM in practice under a clean, controlled experimental setup?

Key findings

• DAC achieves strictly smaller generalization error than DA-ERM in linear regression with label-invariant augmentations, demonstrating intrinsic sample efficiency.
• DAC better tolerates misspecified augmentations, incurring smaller approximation error compared to DA-ERM, which is sensitive to label changes under augmentation.
• Generalization bounds for DAC in two-layer ReLU networks show that consistency regularization reduces model complexity and improves generalization.
• Empirical results on CIFAR-100 with WideResNet confirm that DAC outperforms DA-ERM across all labeled data regimes (1k, 10k, 20k samples), with consistent accuracy gains.
• Theoretical analysis explains why stronger augmentations can be used in consistency regularization (e.g., FixMatch) but not in pseudo-labeling (e.g., FixMatch), due to DAC’s robustness to label shift.
• Experiments show that DAC maintains superior performance even with minimal labeled data, highlighting its value in low-data regimes.