[Paper Review] Micro-Batch Training with Batch-Channel Normalization and Weight Standardization

Siyuan Qiao, Huiyu Wang+3 more|arXiv (Cornell University)|Mar 25, 2019

Intravenous Infusion Technology and Safety60 references123 citations

TL;DR

The paper introduces Weight Standardization (WS) and Batch-Channel Normalization (BCN) to enable effective micro-batch training, showing theoretical smoothing of the loss landscape and empirical gains across vision tasks. WS and BCN aim to replicate BN-like benefits without large batch sizes.

ABSTRACT

Batch Normalization (BN) has become an out-of-box technique to improve deep network training. However, its effectiveness is limited for micro-batch training, i.e., each GPU typically has only 1-2 images for training, which is inevitable for many computer vision tasks, e.g., object detection and semantic segmentation, constrained by memory consumption. To address this issue, we propose Weight Standardization (WS) and Batch-Channel Normalization (BCN) to bring two success factors of BN into micro-batch training: 1) the smoothing effects on the loss landscape and 2) the ability to avoid harmful elimination singularities along the training trajectory. WS standardizes the weights in convolutional layers to smooth the loss landscape by reducing the Lipschitz constants of the loss and the gradients; BCN combines batch and channel normalizations and leverages estimated statistics of the activations in convolutional layers to keep networks away from elimination singularities. We validate WS and BCN on comprehensive computer vision tasks, including image classification, object detection, instance segmentation, video recognition and semantic segmentation. All experimental results consistently show that WS and BCN improve micro-batch training significantly. Moreover, using WS and BCN with micro-batch training is even able to match or outperform the performances of BN with large-batch training.

Motivation & Objective

• Motivate the need for normalization techniques that work well with micro-batch training (1-2 images per GPU).
• Bridge BN-like benefits (loss landscape smoothing and avoidance of elimination singularities) to micro-batch regimes.
• Propose WS to standardize convolutional weights and BCN to combine batch and channel statistics, improving training stability and performance.
• Evaluate WS and BCN across diverse computer vision tasks to validate practical gains.

Proposed method

• Propose Weight Standardization (WS): reparameterize conv weights as WS(W) where W is standardized per output channel to zero mean and unit variance.
• Introduce Batch-Channel Normalization (BCN): combine batch statistics with channel-wise statistics to estimate activations’ means and variances.
• Provide theoretical analysis showing WS reduces Lipschitz constants of the loss and gradients, thereby smoothing the loss landscape.
• Analyze elimination singularities and show BN uniquely keeps activations away from such singularities; argue WS/BCN extend similar properties to micro-batch settings.
• Compare WS with Weight Normalization (WN) and Centered Weight Normalization (CWN).
• Demonstrate WS+BCN can perform comparably or better than BN with large batches and GN with micro-batches across tasks.

Experimental results

Research questions

• RQ1 Can WS and BCN reproduce BN's benefits (landscape smoothing and avoidance of elimination singularities) in micro-batch training?
• RQ2 Do WS and BCN improve training speed and final accuracy across diverse vision tasks with small batch sizes?
• RQ3 How do WS and BCN compare to existing normalization methods (GN/LN) and to BN with large batches?
• RQ4 What is the theoretical impact of WS on Lipschitz constants and elimination singularities?
• RQ5 Are WS and BCN effective when followed by standard normalization layers in common CNN architectures?

Key findings

• WS lowers the Lipschitz constants of the loss and gradients, smoothing the optimization landscape.
• WS and BCN help push networks away from elimination singularities, improving training stability.
• GN+WS with micro-batch training can match or outperform BN with large batches on select tasks.
• BCN provides additional performance gains over GN or BN in both large-batch and micro-batch settings.
• Empirical evaluation covers image classification, object detection, instance segmentation, video recognition, and semantic segmentation, with consistent improvements using WS and BCN.