[Paper Review] Online Normalization for Training Neural Networks

Vitaliy Chiley, Ilya Sharapov+6 more|arXiv (Cornell University)|May 1, 2019

Neural Networks and Applications23 citations

TL;DR

This paper introduces Online Normalization, a technique that normalizes hidden activations in neural networks without requiring batches, using an unbiased gradient computation method compatible with automatic differentiation. It achieves performance on par with Batch Normalization across image classification, segmentation, and language modeling tasks, enabling training in scenarios where batching is impractical.

ABSTRACT

Online Normalization is a new technique for normalizing the hidden activations of a neural network. Like Batch Normalization, it normalizes the sample dimension. While Online Normalization does not use batches, it is as accurate as Batch Normalization. We resolve a theoretical limitation of Batch Normalization by introducing an unbiased technique for computing the gradient of normalized activations. Online Normalization works with automatic differentiation by adding statistical normalization as a primitive. This technique can be used in cases not covered by some other normalizers, such as recurrent networks, fully connected networks, and networks with activation memory requirements prohibitive for batching. We show its applications to image classification, image segmentation, and language modeling. We present formal proofs and experimental results on ImageNet, CIFAR, and PTB datasets.

Motivation & Objective

• Address the theoretical limitations of Batch Normalization, particularly its reliance on mini-batches and biased gradient computation.
• Enable effective normalization in architectures where batching is infeasible, such as recurrent networks and fully connected networks with high memory demands.
• Develop a normalization technique compatible with automatic differentiation by integrating statistical normalization as a primitive operation.
• Achieve training stability and accuracy comparable to Batch Normalization without requiring batch statistics.
• Extend normalization applicability to sequential and memory-intensive deep learning models.

Proposed method

• Introduce a novel, unbiased gradient computation technique for normalized activations, overcoming a key theoretical flaw in Batch Normalization.
• Formalize normalization as a primitive operation within automatic differentiation frameworks, enabling end-to-end training with gradient flow.
• Compute batch statistics online during training using a running estimate of mean and variance, avoiding dependency on fixed mini-batches.
• Apply normalization to the sample dimension of hidden activations, similar to Batch Normalization, but without requiring batch-level statistics.
• Use a recursive update rule for running statistics to maintain accurate normalization during training with sequential or variable-length inputs.
• Integrate the normalization layer as a differentiable component that can be backpropagated through without introducing gradient bias.

Experimental results

Research questions

• RQ1 Can a normalization technique be developed that avoids the batch dependency of Batch Normalization while maintaining comparable performance?
• RQ2 Is it possible to compute unbiased gradients for normalized activations in a way compatible with automatic differentiation?
• RQ3 Can Online Normalization effectively stabilize training in recurrent and fully connected networks where batching is impractical?
• RQ4 How does Online Normalization compare to Batch Normalization in terms of accuracy and convergence across diverse tasks?
• RQ5 Does Online Normalization generalize across different architectures and datasets, including image and sequence modeling tasks?

Key findings

• Online Normalization achieves test accuracy on ImageNet that matches or exceeds that of Batch Normalization, demonstrating parity in performance.
• The method enables stable training of recurrent networks and fully connected networks where batch normalization is infeasible due to memory or sequence constraints.
• Experiments on CIFAR-10 and CIFAR-100 show that Online Normalization maintains high accuracy without requiring batch statistics.
• On the PTB language modeling dataset, Online Normalization achieves competitive perplexity scores, confirming its effectiveness in sequential modeling.
• Theoretical analysis confirms that Online Normalization provides an unbiased gradient estimate, resolving a key limitation of Batch Normalization.
• The technique is compatible with automatic differentiation frameworks, enabling seamless integration into modern deep learning pipelines.