[Paper Review] Norm matters: efficient and accurate normalization schemes in deep networks

Elad Hoffer, Ron Banner+2 more|arXiv (Cornell University)|Mar 5, 2018

Model Reduction and Neural Networks43 references52 citations

TL;DR

The paper reconsiders normalization in deep nets, showing weight decay, learning rate, and normalization interact via weight norms; it proposes L1 and L∞ based BN variants and bounded weight normalization to improve stability and enable half-precision training, with competitive accuracy.

ABSTRACT

Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present a novel view on the purpose and function of normalization methods and weight-decay, as tools to decouple weights' norm from the underlying optimized objective. This property highlights the connection between practices such as normalization, weight decay and learning-rate adjustments. We suggest several alternatives to the widely used $L^2$ batch-norm, using normalization in $L^1$ and $L^\infty$ spaces that can substantially improve numerical stability in low-precision implementations as well as provide computational and memory benefits. We demonstrate that such methods enable the first batch-norm alternative to work for half-precision implementations. Finally, we suggest a modification to weight-normalization, which improves its performance on large-scale tasks.

Motivation & Objective

• Understand how weight norms interact with normalization and learning dynamics.
• Propose normalization alternatives that decouple weight norms from the optimization objective.
• Improve numerical stability and efficiency, especially in low-precision settings.
• Evaluate L1 and L∞ based normalizations as replacements or complements to Batch Normalization.
• Introduce bounded weight normalization to enhance large-scale training performance.

Proposed method

• Treat BN’s invariance to weight norm as a mechanism to decouple scale from optimization.
• Derive and test learning-rate corrections that mimic weight-decay effects on training dynamics.
• Replace or augment L2 BN with L1 and L∞ based BN variants and derive appropriate scaling constants (e.g., C_L1) for stability and performance.
• Demonstrate that L1 BN supports half-precision training where L2 BN may fail.
• Introduce bounded weight normalization (BWN) by fixing channel-wise weight norms to a scalar ρ to improve stability and performance on ImageNet and in seq2seq tasks.
• Explore Lp-weight normalization (including L1 and L∞ variants) as alternatives to standard weight normalization.

Experimental results

Research questions

• RQ1 How does weight norm interact with Batch Normalization to affect learning dynamics and effective step size?
• RQ2 Can alternative norm-based normalizations (L1, L∞) match BN accuracy while offering computational and low-precision benefits?
• RQ3 Does bounding weight norms (bounded weight normalization) improve performance on large-scale tasks and sequence models?
• RQ4 Is it feasible to perform batch normalization in half-precision using L1 normalization?
• RQ5 What are the trade-offs of Lp-weight normalization versus traditional weight normalization?

Key findings

• Weight decay improves optimization by constraining the weight norm, effectively stabilizing the learning rate; similar effects can be achieved by adjusting learning rate or normalization.
• L1 and L∞ based batch normalization can match or closely approach L2 BN accuracy on CIFAR and ImageNet, with L1 BN enabling stable half-precision training.
• L1 BN offers computational and memory advantages and remains robust under quantization noise, enabling half-precision BN where L2 BN fails.
• A bounded weight normalization (BWN) approach significantly improves large-scale task performance (ImageNet) compared to standard weight-normalization, approaching BN performance.
• L1 and Lp normalizations can serve as viable low-precision friendly alternatives to BN across architectures (ResNet, Transformer) with minimal accuracy loss.