[论文解读] Towards Understanding Regularization in Batch Normalization

Ping Luo, Xinjiang Wang+2 more|arXiv (Cornell University)|Sep 4, 2018

Neural Networks and Applications被引用 53

一句话总结

本文将批归一化（BN）视为一种隐式正则化器，它分解为总体归一化（PN）和 γ 衰减，表明 BN 允许更大的学习率并提高泛化性能，在卷积神经网络（CNNs）中有理论和经验支持。

ABSTRACT

Batch Normalization (BN) improves both convergence and generalization in training neural networks. This work understands these phenomena theoretically. We analyze BN by using a basic block of neural networks, consisting of a kernel layer, a BN layer, and a nonlinear activation function. This basic network helps us understand the impacts of BN in three aspects. First, by viewing BN as an implicit regularizer, BN can be decomposed into population normalization (PN) and gamma decay as an explicit regularization. Second, learning dynamics of BN and the regularization show that training converged with large maximum and effective learning rate. Third, generalization of BN is explored by using statistical mechanics. Experiments demonstrate that BN in convolutional neural networks share the same traits of regularization as the above analyses.

研究动机与目标

• Motivate a theoretical understanding of how BN regularizes learning and generalization.
• Decompose BN into population normalization (PN) and gamma decay to characterize explicit regularization.
• Analyze learning dynamics and convergence under BN using ordinary differential equations.
• Compare BN with weight normalization and vanilla SGD through a teacher-student and statistical mechanics framework.
• Validate theoretical insights with CNN experiments on CIFAR-10 and ablation studies.

提出的方法

• Model BN in a single-layer perceptron with ReLU to isolate effects of BN.
• Treat batch statistics as random variables with Gaussian priors to derive a regularization form.
• Decompose BN into PN and gamma decay, yielding a data-dependent regularization strength \u0003(h) for the scale parameter \u0003.
• Use ordinary differential equations to study learning dynamics and derive maximum and effective learning rates.
• Use a teacher-student statistical mechanics framework to analyze generalization under BN, weight normalization, and SGD.
• Empirically validate BN’s regularization properties in CNNs on CIFAR-10 and explore PN+gamma decay as an approximation.

实验结果

研究问题

• RQ1 How can BN be expressed as an explicit regularization in terms of PN and gamma decay?
• RQ2 What are the effects of BN on learning dynamics and the permissible learning rates compared to WN and SGD?
• RQ3 How does BN influence generalization in a teacher-student setting and in CNNs?
• RQ4 What is the role of batch size in BN’s regularization strength and training dynamics?
• RQ5 Can PN+gamma decay approximate BN in practice and how does it compare empirically?

主要发现

• BN can be decomposed into population normalization and gamma decay, with a data-dependent regularization strength on the scale parameter gamma.
• The gamma decay term is adaptive via a factor zeta(h) and depends on batch kurtosis and Fisher information, tying BN’s noise to training dynamics.
• BN enables larger maximum and effective learning rates, leading to faster convergence than SGD or weight normalization in the analyzed model.
• In a large-scale (P, N) regime, BN and WN+gamma decay can yield comparable generalization benefits, with BN often outperforming vanilla SGD.
• Experiments in CNNs show BN shares regularization traits with the theoretical BN model, and PN+gamma decay can mimic BN’s effects under appropriate conditions.