[Paper Review] Towards Understanding Regularization in Batch Normalization

Ping Luo, Xinjiang Wang+2 more|arXiv (Cornell University)|Sep 4, 2018

Neural Networks and ApplicationsComputer Science53 citations

TL;DR

This paper analyzes Batch Normalization (BN) as an implicit regularizer that decomposes into population normalization and gamma decay, showing BN enables larger learning rates and improves generalization, with theoretical and empirical support in CNNs.

ABSTRACT

Batch Normalization (BN) improves both convergence and generalization in training neural networks. This work understands these phenomena theoretically. We analyze BN by using a basic block of neural networks, consisting of a kernel layer, a BN layer, and a nonlinear activation function. This basic network helps us understand the impacts of BN in three aspects. First, by viewing BN as an implicit regularizer, BN can be decomposed into population normalization (PN) and gamma decay as an explicit regularization. Second, learning dynamics of BN and the regularization show that training converged with large maximum and effective learning rate. Third, generalization of BN is explored by using statistical mechanics. Experiments demonstrate that BN in convolutional neural networks share the same traits of regularization as the above analyses.

Motivation & Objective

• Motivate a theoretical understanding of how BN regularizes learning and generalization.
• Decompose BN into population normalization (PN) and gamma decay to characterize explicit regularization.
• Analyze learning dynamics and convergence under BN using ordinary differential equations.
• Compare BN with weight normalization and vanilla SGD through a teacher-student and statistical mechanics framework.
• Validate theoretical insights with CNN experiments on CIFAR-10 and ablation studies.

Proposed method

• Model BN in a single-layer perceptron with ReLU to isolate effects of BN.
• Treat batch statistics as random variables with Gaussian priors to derive a regularization form.
• Decompose BN into PN and gamma decay, yielding a data-dependent regularization strength (h) for the scale parameter .
• Use ordinary differential equations to study learning dynamics and derive maximum and effective learning rates.
• Use a teacher-student statistical mechanics framework to analyze generalization under BN, weight normalization, and SGD.
• Empirically validate BN’s regularization properties in CNNs on CIFAR-10 and explore PN+gamma decay as an approximation.

Experimental results

Research questions

• RQ1 How can BN be expressed as an explicit regularization in terms of PN and gamma decay?
• RQ2 What are the effects of BN on learning dynamics and the permissible learning rates compared to WN and SGD?
• RQ3 How does BN influence generalization in a teacher-student setting and in CNNs?
• RQ4 What is the role of batch size in BN’s regularization strength and training dynamics?
• RQ5 Can PN+gamma decay approximate BN in practice and how does it compare empirically?

Key findings

• BN can be decomposed into population normalization and gamma decay, with a data-dependent regularization strength on the scale parameter gamma.
• The gamma decay term is adaptive via a factor zeta(h) and depends on batch kurtosis and Fisher information, tying BN’s noise to training dynamics.
• BN enables larger maximum and effective learning rates, leading to faster convergence than SGD or weight normalization in the analyzed model.
• In a large-scale (P, N) regime, BN and WN+gamma decay can yield comparable generalization benefits, with BN often outperforming vanilla SGD.
• Experiments in CNNs show BN shares regularization traits with the theoretical BN model, and PN+gamma decay can mimic BN’s effects under appropriate conditions.