[Paper Review] Differentiable Learning-to-Normalize via Switchable Normalization

Ping Luo, Jiamin Ren+3 more|arXiv (Cornell University)|Jun 28, 2018

Domain Adaptation and Few-Shot Learning38 references146 citations

TL;DR

Switchable Normalization (SN) learns to selectively combine three normalization statistics (IN, LN, BN) for each layer using learnable importance weights, yielding robustness to small batch sizes and improved performance across vision tasks.

ABSTRACT

We address a learning-to-normalize problem by proposing Switchable Normalization (SN), which learns to select different normalizers for different normalization layers of a deep neural network. SN employs three distinct scopes to compute statistics (means and variances) including a channel, a layer, and a minibatch. SN switches between them by learning their importance weights in an end-to-end manner. It has several good properties. First, it adapts to various network architectures and tasks (see Fig.1). Second, it is robust to a wide range of batch sizes, maintaining high performance even when small minibatch is presented (e.g. 2 images/GPU). Third, SN does not have sensitive hyper-parameter, unlike group normalization that searches the number of groups as a hyper-parameter. Without bells and whistles, SN outperforms its counterparts on various challenging benchmarks, such as ImageNet, COCO, CityScapes, ADE20K, and Kinetics. Analyses of SN are also presented. We hope SN will help ease the usage and understand the normalization techniques in deep learning. The code of SN has been made available in https://github.com/switchablenorms/.

Motivation & Objective

• Motivate learning-to-normalize by allowing different normalizers across layers rather than a fixed choice.
• Develop a differentiable mechanism to weigh IN, LN, and BN statistics end-to-end.
• Demonstrate robustness of SN to a wide range of minibatch sizes.
• Show that SN adapts to various architectures and tasks without sensitive hyper-parameters.

Proposed method

• Introduce SN that combines three statistics sets: IN, LN, BN, via a weighted average of means and variances (Eq. 3).
• Compute statistics efficiently by reusing computations across IN, LN, BN (Eq. 4).
• Learn importance weights w_k and w_k′ for means and variances through a softmax over control parameters λ_k and λ_k′ (Eq. 5).
• Train jointly the network parameters Θ and the control parameters Φ by back-propagation to minimize the loss L(Θ,Φ).
• Provide geometric interpretation by relating SN to weight normalization to compare normalizers (Remark 1).
• Discuss variants such as sparsity and group SN as future work, and outline inference procedure using batch-averaged statistics.

Experimental results

Research questions

• RQ1 Can a single normalization layer learn to select the most appropriate normalizer for a given layer and task?
• RQ2 Does mixing IN, LN, and BN statistically improve performance and stability across varying minibatch sizes?
• RQ3 Is SN robust to different network architectures and datasets without sensitive hyper-parameters?
• RQ4 How do the learned normalizer weights adapt across tasks like classification, detection, segmentation, and video recognition?

Key findings

• SN outperforms BN and GN across multiple batch settings on ImageNet with ResNet50 (e.g., SN achieves 76.9% top-1 vs BN/GN baselines).
• SN maintains high performance when minibatch sizes are small, unlike BN which degrades significantly; SN tracks or exceeds ideal BN across batch settings.
• SN adapts its normalizer composition per task and dataset, e.g., BN is preferred with larger minibatches while LN dominates with very small minibatches.
• SN improves performance in object detection (Faster R-CNN, Mask R-CNN) and semantic segmentation (Cityscapes, ADE20K), often surpassing GN and SyncBN baselines.
• SN demonstrates competitive or superior results in Kinetics video recognition and other tasks, indicating broad applicability.