[论文解读] Data augmentation instead of explicit regularization

Álex Hernández-García, Peter König|arXiv (Cornell University)|Jun 11, 2018

Advanced Neural Network Applications参考文献 73被引用 81

一句话总结

本文认为数据增强可以达到或超过深度网络中显式正则化（权重衰减和 dropout）的效果，并且在不进行超参数调优的情况下，通常可以实现更好的泛化。

ABSTRACT

Contrary to most machine learning models, modern deep artificial neural networks typically include multiple components that contribute to regularization. Despite the fact that some (explicit) regularization techniques, such as weight decay and dropout, require costly fine-tuning of sensitive hyperparameters, the interplay between them and other elements that provide implicit regularization is not well understood yet. Shedding light upon these interactions is key to efficiently using computational resources and may contribute to solving the puzzle of generalization in deep learning. Here, we first provide formal definitions of explicit and implicit regularization that help understand essential differences between techniques. Second, we contrast data augmentation with weight decay and dropout. Our results show that visual object categorization models trained with data augmentation alone achieve the same performance or higher than models trained also with weight decay and dropout, as is common practice. We conclude that the contribution on generalization of weight decay and dropout is not only superfluous when sufficient implicit regularization is provided, but also such techniques can dramatically deteriorate the performance if the hyperparameters are not carefully tuned for the architecture and data set. In contrast, data augmentation systematically provides large generalization gains and does not require hyperparameter re-tuning. In view of our results, we suggest to optimize neural networks without weight decay and dropout to save computational resources, hence carbon emissions, and focus more on data augmentation and other inductive biases to improve performance and robustness.

研究动机与目标

• Define explicit versus implicit regularization with formal clarity.
• Theoretically compare data augmentation to explicit regularizers under statistical learning theory.
• Empirically evaluate models with/without explicit regularization across benchmarks and architectures.
• Assess adaptability to reduced data and architectural changes.
• Discuss practical implications for training efficiency and generalization.

提出的方法

• Provide formal definitions of explicit and implicit regularization based on representational vs. effective capacity.
• Theoretically discuss generalization bounds (Rademacher complexity) and how augmentation acts as implicit regularization.
• Empirically train All-CNN, WRN, and DenseNet on ImageNet, CIFAR-10, CIFAR-100 with/without weight decay and dropout and with no, light, and heavier data augmentation.
• Measure performance and perform bootstrap analyses to compare gains from augmentation vs. explicit regularization.
• Evaluate robustness when training data is reduced (50% and 10%).
• Analyze training dynamics and data-efficiency under different augmentation regimes.

实验结果

研究问题

• RQ1 Does data augmentation alone provide equal or superior generalization compared to explicit regularizers like weight decay and dropout?
• RQ2 How do augmentation levels (none, light, heavier) influence performance across networks and datasets?
• RQ3 Is explicit regularization more or less beneficial when training data is limited or when architectures change?
• RQ4 What are the training dynamics and resource implications of using augmentation versus explicit regularization?

主要发现

Augmentation Level	No explicit reg.	Weight decay & dropout
None	baseline	3.02 (1.65)
Light	8.46 (3.80)	7.88 (2.60)
Heavier	8.68 (4.69)	7.92 (4.03)

• Data augmentation alone can achieve the same or higher accuracy than models trained with weight decay and dropout in multiple experiments.
• On average, augmentation alone improved accuracy by 8.57% over baseline, while augmentation plus explicit regularization improved by 7.90%.
• In several cases, removing weight decay and dropout yielded state-of-the-art results without tuning hyperparameters, indicating implicit regularization from augmentation and optimization suffices.
• Regularizers like weight decay and dropout show smaller generalization gains and can hinder performance if hyperparameters are not carefully tuned, especially under data scarcity.
• Models with augmentation alone train faster and rely less on learning rate schedules, with more consistent results across runs.
• When training data is reduced to 50% or 10%, explicit regularization degrades performance faster than augmentation alone, highlighting augmentation’s data-efficiency advantages.