[Paper Review] Regularization for Deep Learning: A Taxonomy

Jan Kukačka, Vladimir Golkov+1 more|arXiv (Cornell University)|Oct 29, 2017

Domain Adaptation and Few-Shot Learning36 references186 citations

TL;DR

The paper presents a unifying taxonomy of regularization methods in deep learning, categorizing approaches by data, architecture, error terms, regularization terms, and optimization, and offers practical guidance.

ABSTRACT

Regularization is one of the crucial ingredients of deep learning, yet the term regularization has various definitions, and regularization methods are often studied separately from each other. In our work we present a systematic, unifying taxonomy to categorize existing methods. We distinguish methods that affect data, network architectures, error terms, regularization terms, and optimization procedures. We do not provide all details about the listed methods; instead, we present an overview of how the methods can be sorted into meaningful categories and sub-categories. This helps revealing links and fundamental similarities between them. Finally, we include practical recommendations both for users and for developers of new regularization methods.

Motivation & Objective

• Provide a systematic, unifying framework to categorize regularization methods in deep learning.
• Clarify how different regularization techniques relate across data, architecture, loss, and optimization components.
• Offer practical recommendations for researchers and developers designing regularization strategies.
• Highlight connections and common principles among seemingly disparate regularization methods.

Proposed method

• Organize regularization methods into five main axes: data, network architecture, error term, regularization term, and optimization procedures.
• Define a working taxonomy with subcategories to reveal links and fundamental similarities between methods.
• Discuss data-based transformations, augmentation, and target-preserving augmentations as a key regularization mechanism.
• Present a framework for classifying regularizers by dependencies (e.g., on weights, inputs, outputs) and phase (training/test).
• Provide high-level descriptions and categorizations rather than exhaustive implementation details.

Experimental results

Research questions

• RQ1 How can regularization techniques in deep learning be systematically categorized into meaningful groups?
• RQ2 What are the connections and commonalities among methods that are traditionally treated separately (data, architecture, loss, optimization)?
• RQ3 What practical guidance can be given to practitioners and developers for selecting and designing regularization methods?
• RQ4 How can unlabeled data and semi-supervised approaches be integrated within a regularization framework?

Key findings

• Proposes a unified taxonomy that groups regularization methods by data, architecture, error terms, regularization terms, and optimization procedures.
• Shows that many techniques (e.g., data augmentation, dropout, batch normalization) can be viewed under a common data-based regularization framework.
• Demonstrates that regularization can leverage unlabeled data through regularizers independent of targets, enabling semi-supervised approaches.
• Provides practical recommendations for users and developers on selecting and combining regularization techniques.
• Reveals structural links between seemingly disparate methods, clarifying when and why certain techniques regularize generalization.