[论文解读] Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks

Soham De, Samuel Smith|arXiv (Cornell University)|Feb 24, 2020

Adversarial Robustness in Machine Learning参考文献 47被引用 63

一句话总结

这篇论文解释了批量归一化在初始化时将残差块偏向恒等映射，使非常深的 ResNet 成为可能；它还引入 SkipInit 用于在不使用归一化的情况下训练深度残差网络，并研究批量大小如何影响学习率与正则化。

ABSTRACT

Batch normalization dramatically increases the largest trainable depth of residual networks, and this benefit has been crucial to the empirical success of deep residual networks on a wide range of benchmarks. We show that this key benefit arises because, at initialization, batch normalization downscales the residual branch relative to the skip connection, by a normalizing factor on the order of the square root of the network depth. This ensures that, early in training, the function computed by normalized residual blocks in deep networks is close to the identity function (on average). We use this insight to develop a simple initialization scheme that can train deep residual networks without normalization. We also provide a detailed empirical study of residual networks, which clarifies that, although batch normalized networks can be trained with larger learning rates, this effect is only beneficial in specific compute regimes, and has minimal benefits when the batch size is small.

研究动机与目标

• Explain why batch normalization enables training of very deep residual networks.
• Propose an initialization scheme (SkipInit) to train deep ResNets without normalization.
• Empirically study learning-rate benefits across batch sizes for normalized vs. unnormalized networks.
• Quantify regularization effects of batch normalization in residual architectures.
• Compare SkipInit and Fixup with batch normalization on ImageNet.

提出的方法

• Theoretical analysis of variance propagation in residual blocks at initialization for normalized vs. unnormalized branches.
• Introduction of SkipInit: a learnable scalar α on the residual branch end initialized to α (often 0 or 1/√d) to mimic normalization downscaling.
• Empirical training of Wide-ResNets across depths (16–1000 layers) on CIFAR-10 with SGD, momentum, and data augmentation to compare BN, SkipInit, and variants.
• Systematic batch-size experiments (including ghost batch technique) to assess how learning rates interact with batch size for BN, SkipInit, and unnormalized nets.
• Regularization study by varying batch size and introducing Regularized SkipInit (Dropout, biases) to compare generalization.
• ImageNet experiments comparing BN, SkipInit, and Fixup with and without extra regularization.

实验结果

研究问题

• RQ1 Why does batch normalization enable training of very deep residual networks?
• RQ2 Can an initialization like SkipInit replicate the beneficial effect of BN without using normalization?
• RQ3 How do batch size and learning rate interact for BN, SkipInit, and unnormalized nets in residual architectures?
• RQ4 What is the regularization effect of batch normalization compared to SkipInit?
• RQ5 How do SkipInit and Fixup perform on ImageNet relative to BN under varying batch sizes?

主要发现

• Batch normalization downscales the residual branch by a factor on the order of the square root of depth, biasing residual blocks toward the identity at initialization.
• SkipInit can train very deep residual networks without normalization by initializing a residual-scaling parameter (α) to 0 or 1/√d, matching BN’s downscaling effect.
• BN enables larger stable learning rates and better performance at large batch sizes, but gains are minimal at small batch sizes where BN still outperforms SkipInit in test accuracy.
• Regularization benefits of BN are most pronounced at intermediate batch sizes; Regularized SkipInit can surpass BN in small-batch regimes when additional regularization is used.
• On ImageNet, SkipInit with biases or Fixup can match BN at standard batch sizes but BN outperforms at very large batch sizes; with extra regularization, SkipInit-based variants can exceed BN under small-batch settings.
• Empirical results show that the primary benefit of BN is biasing toward the identity (not simply enabling large learning rates).