[Paper Review] Domain Adaptation for Semantic Segmentation via Patch-Wise Contrastive Learning

Weizhe Liu, David Ferstl+4 more|arXiv (Cornell University)|Apr 22, 2021

Domain Adaptation and Few-Shot Learning53 references25 citations

TL;DR

This paper presents a patch-wise contrastive learning approach for unsupervised and semi-supervised domain adaptation in semantic segmentation, avoiding adversarial training and achieving state-of-the-art performance especially with limited target annotations.

ABSTRACT

We introduce a novel approach to unsupervised and semi-supervised domain adaptation for semantic segmentation. Unlike many earlier methods that rely on adversarial learning for feature alignment, we leverage contrastive learning to bridge the domain gap by aligning the features of structurally similar label patches across domains. As a result, the networks are easier to train and deliver better performance. Our approach consistently outperforms state-of-the-art unsupervised and semi-supervised methods on two challenging domain adaptive segmentation tasks, particularly with a small number of target domain annotations. It can also be naturally extended to weakly-supervised domain adaptation, where only a minor drop in accuracy can save up to 75% of annotation cost.

Motivation & Objective

• Bridge the domain gap by aligning cross-domain features via patch-level structural similarity in label space.
• Leverage contrastive learning instead of adversarial objectives for domain alignment.
• Incorporate unlabeled data through pseudo labeling and entropy regularization to improve target-domain performance.
• Demonstrate effectiveness in unsupervised (UDA), semi-supervised (SSDA), and weakly-supervised settings to reduce annotation costs.

Proposed method

• Represent images as patches and learn per-patch latent features via an encoder and a patch-aware latent projector.
• Define a patch-wise semantic disparity D in label space using a pyramid-like labeling histogram (label-space Spatial Pyramid Matching) across three levels, to determine positive/negative patch pairs for contrastive learning.
• Construct a patch-wise contrastive loss L_cont that brings similar patches (low disparity D) from source/target closer and pushes dissimilar patches apart.
• Use a base loss L_base combining supervised cross-entropy on labeled source/target data and an entropy-regularization loss on unlabeled data to reduce uncertainty.
• Optionally generate pseudo labels for unlabeled target samples to form L_self, improving SSDA performance.
• Integrate Fourier Domain Adaptation (FDA) to translate source images toward the target domain before training.
• Train with a staged strategy: first optimize base losses, then jointly optimize with the contrastive and self-training terms.

Experimental results

Research questions

• RQ1 Can patch-level contrastive alignment across domains outperform adversarial feature alignment for semantic segmentation?
• RQ2 How can label-space structural disparity be leveraged to define meaningful cross-domain positive/negative patch pairs?
• RQ3 What is the impact of unlabeled target data and pseudo labeling on SSDA performance in semantic segmentation?
• RQ4 How far can annotation be reduced (weakly supervised) without severely harming performance?

Key findings

• The proposed patch-wise contrastive approach consistently outperforms state-of-the-art methods in both unsupervised and semi-supervised domain adaptation for semantic segmentation.
• In SSDA, the method yields larger gains when target-domain annotations are scarce (e.g., 50–100 images) compared to fully supervised baselines.
• Using FDA for source-to-target translation along with the patch-wise contrastive loss improves domain adaptation performance.
• The method remains easier to train and more stable than adversarial approaches, due to the non-minimax nature of contrastive learning.
• Weakly-supervised scenarios (partial annotations) can still retain strong performance, potentially reducing annotation costs by up to 75% with minor accuracy loss.