[Paper Review] Data augmentation instead of explicit regularization

Álex Hernández-García, Peter König|arXiv (Cornell University)|Jun 11, 2018

Advanced Neural Network ApplicationsComputer Science73 references81 citations

TL;DR

The paper argues that data augmentation can match or surpass explicit regularization (weight decay and dropout) in deep nets, and often enables better generalization without hyperparameter tuning.

ABSTRACT

Contrary to most machine learning models, modern deep artificial neural networks typically include multiple components that contribute to regularization. Despite the fact that some (explicit) regularization techniques, such as weight decay and dropout, require costly fine-tuning of sensitive hyperparameters, the interplay between them and other elements that provide implicit regularization is not well understood yet. Shedding light upon these interactions is key to efficiently using computational resources and may contribute to solving the puzzle of generalization in deep learning. Here, we first provide formal definitions of explicit and implicit regularization that help understand essential differences between techniques. Second, we contrast data augmentation with weight decay and dropout. Our results show that visual object categorization models trained with data augmentation alone achieve the same performance or higher than models trained also with weight decay and dropout, as is common practice. We conclude that the contribution on generalization of weight decay and dropout is not only superfluous when sufficient implicit regularization is provided, but also such techniques can dramatically deteriorate the performance if the hyperparameters are not carefully tuned for the architecture and data set. In contrast, data augmentation systematically provides large generalization gains and does not require hyperparameter re-tuning. In view of our results, we suggest to optimize neural networks without weight decay and dropout to save computational resources, hence carbon emissions, and focus more on data augmentation and other inductive biases to improve performance and robustness.

Motivation & Objective

• Define explicit versus implicit regularization with formal clarity.
• Theoretically compare data augmentation to explicit regularizers under statistical learning theory.
• Empirically evaluate models with/without explicit regularization across benchmarks and architectures.
• Assess adaptability to reduced data and architectural changes.
• Discuss practical implications for training efficiency and generalization.

Proposed method

• Provide formal definitions of explicit and implicit regularization based on representational vs. effective capacity.
• Theoretically discuss generalization bounds (Rademacher complexity) and how augmentation acts as implicit regularization.
• Empirically train All-CNN, WRN, and DenseNet on ImageNet, CIFAR-10, CIFAR-100 with/without weight decay and dropout and with no, light, and heavier data augmentation.
• Measure performance and perform bootstrap analyses to compare gains from augmentation vs. explicit regularization.
• Evaluate robustness when training data is reduced (50% and 10%).
• Analyze training dynamics and data-efficiency under different augmentation regimes.

Experimental results

Research questions

• RQ1 Does data augmentation alone provide equal or superior generalization compared to explicit regularizers like weight decay and dropout?
• RQ2 How do augmentation levels (none, light, heavier) influence performance across networks and datasets?
• RQ3 Is explicit regularization more or less beneficial when training data is limited or when architectures change?
• RQ4 What are the training dynamics and resource implications of using augmentation versus explicit regularization?

Key findings

• Data augmentation alone can achieve the same or higher accuracy than models trained with weight decay and dropout in multiple experiments.
• On average, augmentation alone improved accuracy by 8.57% over baseline, while augmentation plus explicit regularization improved by 7.90%.
• In several cases, removing weight decay and dropout yielded state-of-the-art results without tuning hyperparameters, indicating implicit regularization from augmentation and optimization suffices.
• Regularizers like weight decay and dropout show smaller generalization gains and can hinder performance if hyperparameters are not carefully tuned, especially under data scarcity.
• Models with augmentation alone train faster and rely less on learning rate schedules, with more consistent results across runs.
• When training data is reduced to 50% or 10%, explicit regularization degrades performance faster than augmentation alone, highlighting augmentation’s data-efficiency advantages.