[Paper Review] L2 Regularization versus Batch and Weight Normalization

Twan van Laarhoven|arXiv (Cornell University)|Jun 16, 2017

Advanced Neural Network Applications4 references210 citations

TL;DR

The paper shows L2 regularization has no regularizing effect when used with normalization (Batch, Weight, or Layer Normalization); instead, it changes weight scale and the effective learning rate, explored theoretically and empirically on CIFAR-10.

ABSTRACT

Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regularization has an influence on the scale of weights, and thereby on the effective learning rate. We investigate this dependence, both in theory, and experimentally. We show that popular optimization methods such as ADAM only partially eliminate the influence of normalization on the learning rate. This leads to a discussion on other ways to mitigate this issue.

Motivation & Objective

• Investigate whether L2 regularization provides true regularization when normalization is used.
• Analyze how weight scale affects gradient updates and learning rate under normalization.
• Theoretically derive how weight scaling interacts with common optimizers.
• Experimentally validate the relationship between regularization, weight scale, and learning rate on CIFAR-10.

Proposed method

• Analyze the objective Lλ(w) = L(w) + λ||w||^2 under normalization where y(Xi; αw, γ, β) = y(Xi; w, γ, β).
• Show that Lλ(αw) = Lλα^2(w), implying the regularization term scales weights without changing the function.
• Derive how the gradient ∇yBN and ∇yWN scale with weight scaling and how this leads to an effective learning rate ηeff = η/||w||^2 for various optimizers.
• Provide theoretical expressions for ηeff under SGD, Momentum, RMSProp, ADAM, and Newton/Hessian-based approaches.
• Conduct experimental validation on CIFAR-10 with a 4-layer CNN and Batch Normalization to illustrate the interaction between λ, η, and weight norms.

Experimental results

Research questions

• RQ1 Does L2 regularization regularize the function when Batch/Weight/Layer Normalization are used?
• RQ2 How does the scale of weights affect the effective learning rate across common optimizers under normalization?
• RQ3 Can normalizing weights or choosing specific optimizers mitigate the interaction between λ and learning rate?
• RQ4 What empirical behavior emerges on CIFAR-10 when varying λ and η under Batch Normalization?

Key findings

• L2 regularization does not regularize the function under normalization; it only pushes weights to smaller scale.
• The gradient and thus the effective learning rate depend on the weight scale, leading to ηeff that is inversely proportional to ||w||^2 for several optimizers.
• Under normalization, decreasing weight scale increases the effective learning rate, which can destabilize training unless controlled.
• For SGD and Nesterov momentum, the optimal learning rate tends to scale as η* ∝ 1/λ; for RMSProp and ADAM, η* ∝ 1/√λ.
• Normalizing weights to have unit norm after each update largely removes the λ-dependence of training dynamics.
• Experiments on CIFAR-10 show the diagonals of constant ηeff yield roughly constant test error, supporting the theoretical ηeff relationships.