[论文解读] Understanding Batch Normalization

Johan Björck, Carla P. Gomes+2 more|arXiv (Cornell University)|Jun 1, 2018

Stochastic Gradient Optimization Techniques参考文献 41被引用 40

一句话总结

Batch Normalization 主要 enables training with larger learning rates, which drives faster convergence and better generalization; without BN, gradients and activations can explode with depth, whereas BN keeps activations zero-mean and unit-variance to stabilize training.

ABSTRACT

Batch normalization (BN) is a technique to normalize activations in intermediate layers of deep neural networks. Its tendency to improve accuracy and speed up training have established BN as a favorite technique in deep learning. Yet, despite its enormous success, there remains little consensus on the exact reason and mechanism behind these improvements. In this paper we take a step towards a better understanding of BN, following an empirical approach. We conduct several experiments, and show that BN primarily enables training with larger learning rates, which is the cause for faster convergence and better generalization. For networks without BN we demonstrate how large gradient updates can result in diverging loss and activations growing uncontrollably with network depth, which limits possible learning rates. BN avoids this problem by constantly correcting activations to be zero-mean and of unit standard deviation, which enables larger gradient steps, yields faster convergence and may help bypass sharp local minima. We further show various ways in which gradients and activations of deep unnormalized networks are ill-behaved. We contrast our results against recent findings in random matrix theory, shedding new light on classical initialization schemes and their consequences.

研究动机与目标

• Investigate the mechanisms behind Batch Normalization (BN) benefits beyond the original hypothesis of internal covariate shift.
• Quantify how BN enables larger learning rates and how this contributes to faster convergence and better generalization.
• Examine how unnormalized networks exhibit ill-behaved gradients and activations, especially with depth, compared to BN-enabled networks.
• Relate empirical findings to random matrix theory insights on initialization and conditioning in deep networks.

提出的方法

• Empirical analysis using a 110-layer ResNet on CIFAR-10 to compare BN vs. no BN under varying learning rates.
• Systematic exploration of learning-rate intervals and training dynamics to identify divergence and stability properties.
• Visualization and measurement of gradient and activation distributions, including means/variances across layers.
• Analysis of convolutional weight gradients and channel-wise influence to understand how BN changes gradient magnitudes.
• Connection to random matrix theory to interpret initialization and conditioning effects on deep networks.

实验结果

研究问题

• RQ1 Does Batch Normalization primarily enable larger learning rates, and is this the main source of its benefits?
• RQ2 How do gradients and activations behave in unnormalized networks vs. BN-enabled networks, particularly with increasing depth?
• RQ3 What is the role of network initialization and conditioning in BN’s effectiveness, in light of random matrix theory?
• RQ4 Is normalizing only the final layer as impactful as intermediate BN layers?
• RQ5 How does BN influence divergence risk when applying large gradient updates?

主要发现

• BN allows training with large learning rates, yielding faster convergence and improved generalization compared to unnormalized networks.
• Without BN, gradients and activations diverge and grow with depth when learning rates are large, whereas BN clamps activations to zero-mean and unit-variance, stabilizing training.
• BN provides robustness to initialization-induced ill-conditioning, aligning with random matrix theory insights about deep linear systems and conditioning.
• Networks with BN show more evenly distributed gradients across classes at initialization, unlike unnormalized networks which exhibit highly correlated, large gradients to a single class.
• A substantial portion of BN’s benefit comes from normalizing the final output layer, which accounts for a large portion of performance gains.