[Paper Review] Source-Free Domain Adaptation for Semantic Segmentation

Yuang Liu, Wei Zhang+1 more|arXiv (Cornell University)|Mar 30, 2021

Domain Adaptation and Few-Shot Learning54 references13 citations

TL;DR

This paper proposes SFDA, a source-free domain adaptation framework for semantic segmentation that transfers knowledge from a pre-trained source model to an unlabeled target domain without access to source data. It uses a generator with dual attention distillation to synthesize source-like fake samples and an intra-domain patch-level self-supervision module to leverage pseudo-labeled patches, achieving performance competitive with state-of-the-art source-driven UDA methods on cross-city and synthetic-to-real benchmarks.

ABSTRACT

Unsupervised Domain Adaptation (UDA) can tackle the challenge that convolutional neural network(CNN)-based approaches for semantic segmentation heavily rely on the pixel-level annotated data, which is labor-intensive. However, existing UDA approaches in this regard inevitably require the full access to source datasets to reduce the gap between the source and target domains during model adaptation, which are impractical in the real scenarios where the source datasets are private, and thus cannot be released along with the well-trained source models. To cope with this issue, we propose a source-free domain adaptation framework for semantic segmentation, namely SFDA, in which only a well-trained source model and an unlabeled target domain dataset are available for adaptation. SFDA not only enables to recover and preserve the source domain knowledge from the source model via knowledge transfer during model adaptation, but also distills valuable information from the target domain for self-supervised learning. The pixel- and patch-level optimization objectives tailored for semantic segmentation are seamlessly integrated in the framework. The extensive experimental results on numerous benchmark datasets highlight the effectiveness of our framework against the existing UDA approaches relying on source data.

Motivation & Objective

• Address the practical limitation of unsupervised domain adaptation (UDA) in semantic segmentation, where source datasets are often unavailable due to privacy or commercial constraints.
• Overcome the challenge of preserving source domain knowledge without access to source data during adaptation.
• Enable effective adaptation using only a pre-trained source model and an unlabeled target domain dataset.
• Design a framework that maintains pixel-level semantic consistency and leverages self-supervised signals from the target domain.
• Close the performance gap between source-free and source-driven UDA methods in semantic segmentation.

Proposed method

• Propose a two-stage framework: knowledge transfer via data-free knowledge distillation using a generator to synthesize fake source-like samples.
• Introduce a dual attention distillation (DAD) mechanism that preserves contextual and semantic relationships in synthesized samples for better pixel-level knowledge transfer.
• Utilize a generator trained to minimize mean absolute error (MAE) between source model predictions and synthesized sample predictions, ensuring distributional alignment.
• Design an intra-domain patch-level self-supervision module (IPSM) that identifies and exploits high-confidence, correctly segmented patches in the target domain for self-supervised learning.
• Optimize the target model using a combination of adversarial loss, MAE loss, DAD loss, and IPSM loss, with adaptive weighting via hyperparameters α and β.
• Apply feature-level distillation between intermediate layers of the source and target models to enhance feature-level consistency during adaptation.

Experimental results

Research questions

• RQ1 Can source domain knowledge be effectively recovered and transferred to a target domain without access to the original source data?
• RQ2 How can dual attention distillation improve the quality of synthesized samples for pixel-level semantic segmentation tasks?
• RQ3 To what extent can patch-level pseudo-labels from the target domain serve as reliable self-supervision signals in the absence of ground-truth annotations?
• RQ4 Does the proposed source-free framework achieve performance comparable to source-driven UDA methods in cross-domain semantic segmentation?
• RQ5 How do hyperparameters α and β affect the balance between reconstruction fidelity and feature-level distillation in the adaptation process?

Key findings

• SFDA achieves a mean IoU of 48.33% on GTA5 → Cityscapes, outperforming the source-only baseline (45.87%) and matching state-of-the-art source-driven UDA methods under source-free conditions.
• On cross-city adaptation (Cityscapes → Rio), SFDA achieves 49.03% mIoU, surpassing the best source-driven baseline (48.74%) in the same setting.
• Abllation studies show that combining DAD and BNS losses improves performance by up to 8.5% over using BNS alone on GTA5 → Cityscapes.
• The IPSM module significantly boosts performance: removing it reduces mIoU by 1.5–2.0% across datasets, confirming its value in exploiting reliable target pseudo-labels.
• Hyperparameter analysis reveals that α ≈ 1.0 and β ≈ 0.5 provide optimal balance between reconstruction and distillation, with performance dropping when weights are too high or too low.
• Visualization confirms that DAD-generated samples preserve semantic structure and context, while BNS loss ensures distributional fidelity to the source domain.