[Paper Review] Unsupervised Domain Adaptation for Semantic Segmentation by Content Transfer

Suhyeon Lee, Junhyuk Hyun+2 more|arXiv (Cornell University)|Dec 23, 2020

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a novel unsupervised domain adaptation method for semantic segmentation that separates image content from style using a zero-style loss, enabling domain-invariant feature learning and improving performance on tail classes via content transfer from synthetic to real domains. The method achieves state-of-the-art mIoU on both GTA5→Cityscapes and SYNTHIA→Cityscapes benchmarks by reducing domain gap and mitigating class imbalance.

ABSTRACT

In this paper, we tackle the unsupervised domain adaptation (UDA) for semantic segmentation, which aims to segment the unlabeled real data using labeled synthetic data. The main problem of UDA for semantic segmentation relies on reducing the domain gap between the real image and synthetic image. To solve this problem, we focused on separating information in an image into content and style. Here, only the content has cues for semantic segmentation, and the style makes the domain gap. Thus, precise separation of content and style in an image leads to effect as supervision of real data even when learning with synthetic data. To make the best of this effect, we propose a zero-style loss. Even though we perfectly extract content for semantic segmentation in the real domain, another main challenge, the class imbalance problem, still exists in UDA for semantic segmentation. We address this problem by transferring the contents of tail classes from synthetic to real domain. Experimental results show that the proposed method achieves the state-of-the-art performance in semantic segmentation on the major two UDA settings.

Motivation & Objective

• Address the domain gap between synthetic and real images in unsupervised domain adaptation (UDA) for semantic segmentation.
• Reduce bias toward head classes caused by class imbalance in real-world datasets.
• Improve segmentation performance on rare (tail) classes in the target domain without requiring additional annotations.
• Enable end-to-end training for domain adaptation and class imbalance mitigation using only synthetic source labels and unlabeled real data.
• Achieve state-of-the-art performance on standard UDA benchmarks for semantic segmentation.

Proposed method

• Introduce a zero-style loss that enforces disentanglement of content and style by using one content encoder and two style encoders.
• Train the content encoder to extract domain-invariant features while style encoders learn domain-specific characteristics.
• Apply cycle-consistency learning to enhance feature alignment between source and target domains.
• Implement input-level and output-level content transfer to inject accurate synthetic annotations of tail classes into the real domain.
• Use self-training with pseudo-labels on the target domain, but mitigate its bias by replacing unreliable pseudo-labels for tail classes with transferred content from the source.
• Optimize the model end-to-end using a combination of adversarial loss, cycle-consistency, and content transfer supervision.

Experimental results

Research questions

• RQ1 Can disentangling content and style in images improve domain alignment for semantic segmentation in unsupervised domain adaptation?
• RQ2 Does zero-style loss lead to better feature-level domain adaptation than standard adversarial or cycle-consistency methods?
• RQ3 Can content transfer from synthetic to real data improve segmentation performance on rare (tail) classes without additional annotations?
• RQ4 How does content transfer compare to self-training with pseudo-labels in terms of mitigating class imbalance in UDA?
• RQ5 Does the proposed method achieve state-of-the-art performance on standard synthetic-to-real UDA benchmarks?

Key findings

• The proposed method achieves a mIoU of 49.6% on GTA5→Cityscapes and 46.5% on SYNTHIA→Cityscapes, outperforming all prior SOTA methods.
• The inclusion of input-level content transfer increases mIoU_tail from 29.9% to 31.5% on GTA5→Cityscapes and from 23.2% to 26.3% on SYNTHIA→Cityscapes.
• Output-level content transfer further improves mIoU_tail to 33.3% and mIoU to 49.6% on GTA5→Cityscapes, and to 27.2% and 46.5% on SYNTHIA→Cityscapes.
• Self-training alone degrades mIoU_tail by 0.7% and 0.8% on GTA5→Cityscapes and SYNTHIA→Cityscapes, respectively, confirming its amplification of class imbalance.
• Qualitative results show improved segmentation of tail-class objects (e.g., traffic signs, bicycles) after content transfer, with clearer boundaries and reduced false positives.
• The ablation study confirms that zero-style loss and content transfer are both essential components, with the strongest performance when used together.