[Paper Review] Source-free Domain Adaptation via Distributional Alignment by Matching Batch Normalization Statistics

Masato Ishii, Masashi Sugiyama|arXiv (Cornell University)|Jan 19, 2021

Domain Adaptation and Few-Shot LearningComputer Science25 references29 citations

TL;DR

A source-free domain adaptation method that fine-tunes only the target encoder to align BN statistics with a fixed classifier, plus information maximization to boost discriminability, achieving competitive results without access to source data.

ABSTRACT

In this paper, we propose a novel domain adaptation method for the source-free setting. In this setting, we cannot access source data during adaptation, while unlabeled target data and a model pretrained with source data are given. Due to lack of source data, we cannot directly match the data distributions between domains unlike typical domain adaptation algorithms. To cope with this problem, we propose utilizing batch normalization statistics stored in the pretrained model to approximate the distribution of unobserved source data. Specifically, we fix the classifier part of the model during adaptation and only fine-tune the remaining feature encoder part so that batch normalization statistics of the features extracted by the encoder match those stored in the fixed classifier. Additionally, we also maximize the mutual information between the features and the classifier's outputs to further boost the classification performance. Experimental results with several benchmark datasets show that our method achieves competitive performance with state-of-the-art domain adaptation methods even though it does not require access to source data.

Motivation & Objective

• Address domain shift without access to source data during adaptation.
• Leverage BN statistics stored in a pretrained model to approximate source feature distribution.
• Fine-tune only the target feature encoder while keeping the classifier fixed.
• Enhance target-domain classification with information maximization.
• Demonstrate competitive performance on standard domain adaptation benchmarks.

Proposed method

• Divide the pretrained model into a fixed classifier and a tunable target encoder.
• Define BN-statistics matching loss using Gaussian approximations to compare target BN stats with stored source BN stats in the classifier.
• Minimize KL divergence between target feature distribution (from BN) and the approximated source distribution via BN statistics.
• Incorporate an information maximization loss to encourage discriminative and diverse target predictions.
• Jointly optimize the encoder parameters to minimize L_IM + lambda * L_BNM on unlabeled target data.
• Argue stability and efficiency of this approach under source-free constraints.

Experimental results

Research questions

• RQ1 Can BN statistics stored in a pretrained classifier effectively approximate the source feature distribution for domain alignment without source data?
• RQ2 Does optimizing BN-statistics alignment together with information maximization improve target-domain classification under source-free adaptation?
• RQ3 How does the proposed method perform on standard domain adaptation benchmarks compared to other source-free and typical DA methods?

Key findings

• The method achieves competitive accuracy with state-of-the-art source-free DA methods on benchmark datasets.
• BN-statistics matching (via KL divergence between Gaussian approximations) reduces distributional discrepancy between domains.
• Information maximization improves discriminability and helps avoid overfitting to target data.
• It performs well across multiple scenarios in Office-31 and on digit recognition tasks, often surpassing some typical DA methods.
• The approach demonstrates stability across a wide range of the hyper-parameter lambda and remains effective with smaller target datasets.