[Paper Review] Iterative Loop Method Combining Active and Semi-Supervised Learning for Domain Adaptive Semantic Segmentation

Licong Guan, Yuan Xue|arXiv (Cornell University)|Jan 31, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes ILM-ASSL, an iterative loop method combining semi-supervised and active learning for domain-adaptive semantic segmentation. It first leverages unlabeled target data via semi-supervised learning to improve model accuracy and reduce noisy pseudo-labels, then applies active learning with prediction uncertainty to select and correct critical samples, achieving state-of-the-art performance with minimal annotation cost—improving mIoU by 4.9% and 5.2% on GTAV→Cityscapes and SYNTHIA→Cityscapes, respectively.

ABSTRACT

Semantic segmentation is an important technique for environment perception in intelligent transportation systems. With the rapid development of convolutional neural networks (CNNs), road scene analysis can usually achieve satisfactory results in the source domain. However, guaranteeing good generalization to different target domain scenarios remains a significant challenge. Recently, semi-supervised learning and active learning have been proposed to alleviate this problem. Semisupervised learning can improve model accuracy with massive unlabeled data, but some pseudo labels containing noise would be generated with limited or imbalanced training data. And there will be suboptimal models if human guidance is absent. Active learning can select more effective data to intervene, while the model accuracy can not be improved because the massive unlabeled data are not used. And the probability of querying sub-optimal samples will increase when the domain difference is too large, increasing annotation cost. This paper proposes an iterative loop method combining active and semisupervised learning for domain adaptive semantic segmentation. The method first uses semi-supervised to learn massive unlabeled data to improve model accuracy and provide more accurate selection models for active learning. Secondly, combined with the predictive uncertainty sample selection strategy of active learning, manual intervention is used to correct the pseudo-labels. Finally, flexible iterative loops achieve the best performance with minimal labeling cost. Extensive experiments show that our method establishes state-of-the-art performance on tasks of GTAV to Cityscapes, SYNTHIA to Cityscapes, improving by 4.9% mIoU and 5.2% mIoU, compared to the previous best method, respectively.

Motivation & Objective

• Address the challenge of poor generalization in semantic segmentation when transferring models from source to target domains with domain shift.
• Reduce the high annotation cost of human-annotated data in real-world target domains while maintaining high model performance.
• Overcome the limitations of standalone semi-supervised learning (noisy pseudo-labels) and active learning (underutilization of unlabeled data).
• Develop a practical, image-level sample selection strategy compatible with existing labeling tools like Labelme, using single images as the smallest selection unit.
• Achieve state-of-the-art performance in domain-adaptive semantic segmentation with minimal human intervention and labeling budget.

Proposed method

• Apply semi-supervised learning (SSL) to leverage large-scale unlabeled target domain data, improving model accuracy and generating more reliable pseudo-labels.
• Integrate active learning with a prediction uncertainty-based sample selection strategy to identify and prioritize the most informative images for human annotation.
• Use human-annotated corrections on uncertain samples to refine pseudo-labels and retrain the model, reducing noise and improving generalization.
• Implement an iterative loop between SSL and active learning, alternating between self-training on unlabeled data and human-in-the-loop refinement of high-uncertainty samples.
• Design a practical, image-level selection mechanism compatible with standard labeling tools like Labelme, avoiding pixel-level selection that is impractical in real-world workflows.
• Utilize DeepLab-V3+ with ResNet-101 backbone and incorporate loss functions for unlabeled data (ℒu) and consistency regularization (ℒc) to stabilize training.

Experimental results

Research questions

• RQ1 Can combining semi-supervised and active learning reduce annotation cost while improving performance in domain-adaptive semantic segmentation?
• RQ2 How does iterative refinement using human-corrected pseudo-labels from uncertain samples affect model generalization across domain shifts?
• RQ3 To what extent can a practical, image-level selection strategy based on prediction uncertainty outperform pixel-level selection in real-world labeling pipelines?
• RQ4 Does the proposed method achieve state-of-the-art performance on standard benchmarks like GTAV→Cityscapes and SYNTHIA→Cityscapes with minimal labeling budget?
• RQ5 How robust is the method under challenging settings such as open-set and source-free domain adaptation?

Key findings

• ILM-ASSL achieves a 78.1% mIoU on the GTAV→Cityscapes benchmark, representing a 4.9% improvement over the previous state-of-the-art method.
• On the SYNTHIA→Cityscapes benchmark, the method achieves 81.1% mIoU, a 5.2% gain over the prior best method.
• The ablation study confirms that each component—semi-supervised learning, consistency regularization, and active learning—contributes significantly, with the full method outperforming the baseline by 10.38%.
• t-SNE visualization shows that ILM-ASSL learns more discriminative and well-separated feature representations compared to baseline methods like RIPU.
• The method generalizes well to source-free domain adaptation, outperforming existing SoTA methods such as URMA, LD, SFDA, and RIPU on challenging benchmarks.
• Even with only 2.2% of the target domain data labeled, ILM-ASSL achieves 76.11% mIoU, demonstrating high labeling efficiency.