[Paper Review] Revisiting Batch Normalization For Practical Domain Adaptation

Yanghao Li, Naiyan Wang+3 more|arXiv (Cornell University)|Mar 15, 2016

Domain Adaptation and Few-Shot Learning38 references367 citations

TL;DR

Adaptive Batch Normalization (AdaBN) adapts a pre-trained BN network to new domains by replacing BN statistics per domain, achieving strong domain adaptation without extra parameters or training requirements.

ABSTRACT

Deep neural networks (DNN) have shown unprecedented success in various computer vision applications such as image classification and object detection. However, it is still a common annoyance during the training phase, that one has to prepare at least thousands of labeled images to fine-tune a network to a specific domain. Recent study (Tommasi et al. 2015) shows that a DNN has strong dependency towards the training dataset, and the learned features cannot be easily transferred to a different but relevant task without fine-tuning. In this paper, we propose a simple yet powerful remedy, called Adaptive Batch Normalization (AdaBN) to increase the generalization ability of a DNN. By modulating the statistics in all Batch Normalization layers across the network, our approach achieves deep adaptation effect for domain adaptation tasks. In contrary to other deep learning domain adaptation methods, our method does not require additional components, and is parameter-free. It archives state-of-the-art performance despite its surprising simplicity. Furthermore, we demonstrate that our method is complementary with other existing methods. Combining AdaBN with existing domain adaptation treatments may further improve model performance.

Motivation & Objective

• Motivate and analyze domain shift in deep networks beyond the output layer.
• Propose a simple, parameter-free adaptation method using Batch Normalization statistics.
• Demonstrate AdaBN on standard domain adaptation benchmarks (Office, Caltech-Bing) and a remote sensing cloud-detection task.
• Show that AdaBN is complementary to other adaptation methods and effective with varying target domain data availability.

Proposed method

• Use domain-specific statistics in Batch Normalization layers to align feature distributions across source and target domains.
• Estimate per-domain BN means and variances online from target-domain data and apply them during testing.
• Keep BN parameters (gamma, beta) learned from source domain unchanged; only switch statistics per domain.
• Optionally extend AdaBN to multi-source and semi-supervised settings by domain-wise statistics.
• Provide empirical analysis of feature divergence and sensitivity to target-domain data size.
• Demonstrate practical applicability on large-scale remote sensing image segmentation.

Experimental results

Research questions

• RQ1 Can domain shift be effectively mitigated by domain-specific BN statistics without changing model weights?
• RQ2 How does AdaBN perform on standard single-source and multi-source domain adaptation benchmarks compared to state-of-the-art methods?
• RQ3 How much target-domain data is needed to obtain stable BN statistics for AdaBN?
• RQ4 Can AdaBN complement other domain adaptation techniques to yield further improvements?
• RQ5 Is AdaBN practical for large-scale, real-world tasks such as remote sensing image analysis?

Key findings

• AdaBN improves single-source domain adaptation, achieving competitive or superior results on standard benchmarks.
• AdaBN remains effective when combined with CORAL, yielding additional gains in some tasks.
• On Office-31, AdaBN plus CORAL reaches 75.4/96.2/99.6/72.7/59.0/60.5/77.2 (A→W, D→W, W→D, A→D, D→A, W→A, Avg).
• In multi-source settings, AdaBN outperforms the baseline and single-method CORAL on average (AdaBN Avg 83.6 vs CORAL 83.3).
• On Caltech-Bing, AdaBN outperforms the baseline and is comparable to or better than Deep CORAL variants in reported setups.
• AdaBN demonstrates significant performance gains in a practical cloud-detection task for remote sensing, with large domain gaps between satellites.