[Paper Review] Normalization Techniques in Training DNNs: Methodology, Analysis and Application

Lei Huang, Jie Qin+4 more|arXiv (Cornell University)|Sep 27, 2020

Advanced Neural Network Applications282 references52 citations

TL;DR

This survey reviews normalization methods for training DNNs, proposes a unified taxonomy, and discusses activations, weights, and gradients normalization with applications and theory.

ABSTRACT

Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs), and have successfully been used in various applications. This paper reviews and comments on the past, present and future of normalization methods in the context of DNN training. We provide a unified picture of the main motivation behind different approaches from the perspective of optimization, and present a taxonomy for understanding the similarities and differences between them. Specifically, we decompose the pipeline of the most representative normalizing activation methods into three components: the normalization area partitioning, normalization operation and normalization representation recovery. In doing so, we provide insight for designing new normalization technique. Finally, we discuss the current progress in understanding normalization methods, and provide a comprehensive review of the applications of normalization for particular tasks, in which it can effectively solve the key issues.

Motivation & Objective

• Motivation to accelerate DNN training and improve generalization through normalization.
• Provide a unified taxonomy to compare diverse normalization methods.
• Analyze how normalization affects optimization and training dynamics.
• Summarize advances in task-tailored normalization methods and their applications.
• Discuss theoretical insights and open questions in normalization understanding.

Proposed method

• Decompose normalization methods into normalization area partitioning (NAP), normalization operation (NOP), and normalization representation recovery (NRR).
• Classify activations-normalization by population statistics versus function-based approaches.
• Review normalization types: activations (population statistics and function-based), weights, and gradients.
• Discuss whitening, decorrelation, and standardization variants with backpropagation considerations.
• Analyze BN, LN, GN, IN, PN, BGN and local normalization methods in a unified framework.
• Summarize optimization-theory perspectives including conditioning and Fisher information approximations.

Experimental results

Research questions

• RQ1 What are the main motivations behind different normalization methods in DNNs and how can a taxonomy capture their similarities and differences?
• RQ2 How can we bridge empirical success of normalization with theoretical understanding?
• RQ3 What recent advances enable task-tailored normalization for diverse applications?
• RQ4 How do normalization methods influence optimization conditioning, stability, and generalization?

Key findings

• Normalization methods aim to equalize statistics of layer inputs/gradients across layers to avoid ill-conditioned optimization landscapes.
• Activations can be normalized by population statistics or as function transforms, with a three-part decomposition (NAP, NOP, NRR).
• BN popularizes mini-batch statistics but faces small-batch and training-inference gaps; LN, GN, IN address these limitations.
• Whitening-based approaches (BW, ZCA, PCA) extend standardization and can improve conditioning, with group-based whitening reducing cost.
• A framework (Algorithm 1) conceptualizes normalization as area partitioning, operation, and representation recovery, guiding new method design.
• Applications discussed include domain adaptation, style transfer, GANs, and efficient models where normalization edits activation statistics.