[Paper Review] Semi-supervised Domain Adaptation based on Dual-level Domain Mixing for Semantic Segmentation

Shuaijun Chen, Xu Jia+3 more|arXiv (Cornell University)|Mar 8, 2021

Domain Adaptation and Few-Shot Learning36 references9 citations

TL;DR

This paper proposes a dual-level domain mixing framework for semi-supervised domain adaptation in semantic segmentation, combining region-level and sample-level data mixing to train complementary teachers that distill knowledge into a student model. The method achieves state-of-the-art performance on GTA5-to-Cityscapes benchmarks, outperforming existing methods even without style transfer, by iteratively refining pseudo labels and progressively improving both teachers and student through self-training.

ABSTRACT

Data-driven based approaches, in spite of great success in many tasks, have poor generalization when applied to unseen image domains, and require expensive cost of annotation especially for dense pixel prediction tasks such as semantic segmentation. Recently, both unsupervised domain adaptation (UDA) from large amounts of synthetic data and semi-supervised learning (SSL) with small set of labeled data have been studied to alleviate this issue. However, there is still a large gap on performance compared to their supervised counterparts. We focus on a more practical setting of semi-supervised domain adaptation (SSDA) where both a small set of labeled target data and large amounts of labeled source data are available. To address the task of SSDA, a novel framework based on dual-level domain mixing is proposed. The proposed framework consists of three stages. First, two kinds of data mixing methods are proposed to reduce domain gap in both region-level and sample-level respectively. We can obtain two complementary domain-mixed teachers based on dual-level mixed data from holistic and partial views respectively. Then, a student model is learned by distilling knowledge from these two teachers. Finally, pseudo labels of unlabeled data are generated in a self-training manner for another few rounds of teachers training. Extensive experimental results have demonstrated the effectiveness of our proposed framework on synthetic-to-real semantic segmentation benchmarks.

Motivation & Objective

• Address the performance gap in semantic segmentation when transferring from synthetic (source) to real-world (target) domains with limited labeled target data.
• Overcome limitations of unsupervised domain adaptation and semi-supervised learning by combining labeled source data, limited labeled target data, and abundant unlabeled target data.
• Reduce domain shift by introducing dual-level data mixing—region-level and sample-level—to encourage domain-invariant feature learning.
• Improve model generalization and robustness through knowledge distillation from complementary teachers and iterative self-training using pseudo-labeled data.
• Demonstrate the effectiveness of the framework on standard synthetic-to-real benchmarks, achieving superior performance with minimal labeled target data.

Proposed method

• Propose region-level domain mixing by applying two masks to labeled source and target images and combining masked regions to create mixed samples that emphasize structural invariance.
• Introduce sample-level domain mixing by directly blending entire source and target images to encourage holistic domain-invariant representation learning.
• Train two complementary teacher models—one from region-level mixed data and one from sample-level mixed data—each focusing on different aspects of domain shift.
• Use knowledge distillation to transfer knowledge from both teachers to a single student model, leveraging 'dark knowledge' to improve generalization on the target domain.
• Generate pseudo labels for unlabeled target data using the student model and retrain the teachers in an iterative self-training loop to progressively refine both teachers and student.
• Apply style transfer in the LAB color space to further reduce visual domain gap, though the method remains effective even without it.

Experimental results

Research questions

• RQ1 Can dual-level domain mixing—operating at both region-level and sample-level—improve domain generalization in semi-supervised domain adaptation for semantic segmentation?
• RQ2 How does knowledge distillation from two complementary teachers trained on different mixing levels affect student model performance compared to single-teacher or baseline methods?
• RQ3 To what extent does iterative self-training with pseudo-labeled data enhance model performance across multiple rounds of refinement?
• RQ4 Does the proposed framework remain effective when style transfer is not applied, and how does it compare to existing methods like ASS under the same conditions?
• RQ5 How do the complementary strengths of region-level and sample-level mixing manifest across different semantic categories, especially those with complex or distinctive shapes?

Key findings

• The student model consistently outperforms both individual teachers across all labeled data settings, demonstrating the effectiveness of knowledge distillation from complementary views.
• On the GTA5-to-Cityscapes benchmark, the proposed method achieves 69.8% mean IoU with 2975 labeled target images, surpassing the ASS baseline even without style transfer.
• The ensemble of models from dual-level mixing achieves better performance than single-view ensembles, with improved IoU on challenging categories like fence and rider.
• Region-level mixing performs best on structure-robust classes (e.g., road, sidewalk, vegetation), while sample-level mixing excels on shape-sensitive classes (e.g., bus, light), confirming their complementary nature.
• The iterative self-training loop leads to progressive improvement: student performance increases across rounds, and stronger students refine teacher models via more accurate pseudo labels.
• Style transfer enhances performance, but the framework remains effective without it—achieving state-of-the-art results on 100–1000 labeled images even in the absence of domain translation.