[Paper Review] Weight Standardization

Siyuan Qiao, Huiyu Wang+3 more|arXiv (Cornell University)|Mar 25, 2019

Advanced Neural Network ApplicationsComputer Science49 citations

TL;DR

This paper proposes Weight Standardization (WS) and Batch-Channel Normalization (BCN) to enhance deep network training under micro-batch settings, where standard Batch Normalization (BN) underperforms due to insufficient batch statistics. WS reduces the Lipschitz constant of loss and gradients by standardizing convolutional layer weights, while BCN combines batch and channel normalization using estimated activation statistics to avoid harmful singularities, significantly improving performance in object detection, segmentation, and video recognition tasks even matching or surpassing BN with large batches.

ABSTRACT

Batch Normalization (BN) has become an out-of-box technique to improve deep network training. However, its effectiveness is limited for micro-batch training, i.e., each GPU typically has only 1-2 images for training, which is inevitable for many computer vision tasks, e.g., object detection and semantic segmentation, constrained by memory consumption. To address this issue, we propose Weight Standardization (WS) and Batch-Channel Normalization (BCN) to bring two success factors of BN into micro-batch training: 1) the smoothing effects on the loss landscape and 2) the ability to avoid harmful elimination singularities along the training trajectory. WS standardizes the weights in convolutional layers to smooth the loss landscape by reducing the Lipschitz constants of the loss and the gradients; BCN combines batch and channel normalizations and leverages estimated statistics of the activations in convolutional layers to keep networks away from elimination singularities. We validate WS and BCN on comprehensive computer vision tasks, including image classification, object detection, instance segmentation, video recognition and semantic segmentation. All experimental results consistently show that WS and BCN improve micro-batch training significantly. Moreover, using WS and BCN with micro-batch training is even able to match or outperform the performances of BN with large-batch training.

Motivation & Objective

• Address the performance degradation of Batch Normalization (BN) in micro-batch training, where each GPU processes only 1–2 images due to memory constraints.
• Overcome the limitations of BN in micro-batch scenarios by introducing techniques that preserve its key benefits: loss landscape smoothing and avoidance of harmful singularities.
• Enable effective training of deep networks in memory-constrained vision tasks such as object detection, instance segmentation, and semantic segmentation using micro-batches.
• Develop methods that maintain training stability and performance without relying on large batch sizes, which are often infeasible in practice.

Proposed method

• Weight Standardization (WS) standardizes the weights of convolutional layers by dividing them by the standard deviation of their weights across channels, reducing the Lipschitz constant of the loss and gradients.
• WS is applied per convolutional layer and does not require batch statistics, making it suitable for micro-batch training.
• Batch-Channel Normalization (BCN) combines batch normalization and channel normalization, using estimated activation statistics to stabilize training and avoid elimination singularities.
• BCN leverages both batch-level and channel-level statistics to maintain normalization stability when batch size is small.
• The proposed methods are integrated into standard training pipelines and applied across diverse vision tasks without architectural changes.
• WS and BCN are designed to be plug-and-play, compatible with existing models and optimizers.

Experimental results

Research questions

• RQ1 Can weight standardization effectively smooth the loss landscape in micro-batch training, where BN fails due to insufficient statistics?
• RQ2 Can combining batch and channel normalization mitigate harmful singularities in the training trajectory under micro-batch conditions?
• RQ3 Does the combination of WS and BCN enable training performance comparable to BN with large-batch training in memory-constrained vision tasks?
• RQ4 Can WS and BCN be universally applied across diverse computer vision tasks without architectural modifications?

Key findings

• Weight Standardization significantly reduces the Lipschitz constant of the loss and gradients, leading to a smoother loss landscape in micro-batch training.
• Batch-Channel Normalization effectively prevents the network from collapsing into harmful elimination singularities during training with small batches.
• The combination of WS and BCN enables consistent performance improvements across image classification, object detection, instance segmentation, video recognition, and semantic segmentation.
• Micro-batch training with WS and BCN matches or exceeds the performance of BN with large-batch training on all evaluated tasks.
• The proposed methods are robust and generalizable, delivering gains without requiring changes to model architecture or training hyperparameters.