[Paper Review] Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe, Christian Szegedy|arXiv (Cornell University)|Feb 11, 2015

Neural Networks and Applications23 references24,246 citations

TL;DR

The paper introduces Batch Normalization, a method that normalizes layer inputs within mini-batches to reduce internal covariate shift, enabling higher learning rates, regularization, and faster training, achieving state-of-the-art ImageNet results.

ABSTRACT

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of human raters.

Motivation & Objective

• Motivate the problem of internal covariate shift in deep networks during training.
• Propose a normalization technique integrated into the network architecture that operates on mini-batches.
• Show that BN enables higher learning rates and acts as a regularizer, reducing or removing the need for Dropout.
• Demonstrate accelerated training and improved accuracy on large-scale vision tasks (ImageNet) using BN.
• Provide practical guidelines for training and inference with batch-normalized networks.

Proposed method

• Insert a Batch Normalization transform before nonlinearities to normalize each activation dimension to zero mean and unit variance using mini-batch statistics.
• Learn per-dimension scale (gamma) and shift (beta) parameters to preserve network representational capacity.
• Backpropagate through the BN transform to update gamma, beta, and earlier layer parameters.
• During inference, use population statistics (or their moving averages) instead of mini-batch statistics for deterministic outputs.
• Apply BN to convolutional networks by normalizing feature maps across batch and spatial locations (per feature map).
• Demonstrate training with higher learning rates, less sensitivity to initialization, and reduced need for Dropout.

Experimental results

Research questions

• RQ1 Does integrating batch-wise normalization reduce internal covariate shift and accelerate training of deep networks?
• RQ2 Can BN enable higher learning rates without divergence and improve gradient flow across layers?
• RQ3 What impact does BN have on regularization and generalization, compared to or in combination with Dropout?
• RQ4 How does BN affect performance on large-scale vision tasks like ImageNet, including single-network and ensemble results?

Key findings

• Batch Normalization enables much higher learning rates and reduces sensitivity to parameter initialization.
• Networks with BN converge faster and can achieve the same or better accuracy with substantially fewer training steps (e.g., 14x fewer steps to reach a given accuracy on ImageNet variants).
• BN achieves state-of-the-art results on ImageNet, with an ensemble reaching 4.9% top-5 validation error (and 4.8% test error).
• BN-Baseline matches Inception's accuracy in less than half the training steps, and further BN variants reach higher final accuracy (e.g., 74.8% top-5 on validation with BN-x30).
• Batch Normalization reduces or eliminates the need for Dropout in some settings and can stabilize training when using saturating nonlinearities like sigmoid.
• BN improves gradient propagation by making layer Jacobians less sensitive to parameter scale and may regularize the model.