[Paper Review] Adaptive Semantic Segmentation with a Strategic Curriculum of Proxy Labels

Kashyap Chitta, Jianwei Feng+1 more|arXiv (Cornell University)|Nov 8, 2018

Topic Modeling27 references15 citations

TL;DR

This paper proposes a strategic curriculum for unsupervised domain adaptation in semantic segmentation, using proxy labels from the target domain and combining easy mining from proxy-labeled target data with hard mining from labeled source data. By training with a dynamically balanced mix of these samples and using a dual-decoder architecture to assess prediction confidence, the method achieves state-of-the-art performance on synthetic-to-real adaptation benchmarks without adversarial feature alignment.

ABSTRACT

Training deep networks for semantic segmentation requires annotation of large amounts of data, which can be time-consuming and expensive. Unfortunately, these trained networks still generalize poorly when tested in domains not consistent with the training data. In this paper, we show that by carefully presenting a mixture of labeled source domain and proxy-labeled target domain data to a network, we can achieve state-of-the-art unsupervised domain adaptation results. With our design, the network progressively learns features specific to the target domain using annotation from only the source domain. We generate proxy labels for the target domain using the network's own predictions. Our architecture then allows selective mining of easy samples from this set of proxy labels, and hard samples from the annotated source domain. We conduct a series of experiments with the GTA5, Cityscapes and BDD100k datasets on synthetic-to-real domain adaptation and geographic domain adaptation, showing the advantages of our method over baselines and existing approaches.

Motivation & Objective

• To address domain shift in semantic segmentation where models trained on source domains generalize poorly to target domains due to distribution shifts.
• To reduce reliance on expensive pixel-level annotations in target domains by leveraging unlabeled target data through proxy labeling.
• To improve unsupervised domain adaptation performance through a curriculum-based training strategy that balances easy and hard samples from source and target domains.
• To develop a lightweight, non-adversarial architecture that enables effective sample difficulty estimation via ensemble agreement maps.
• To demonstrate superior performance on synthetic-to-real and cross-city domain adaptation benchmarks without explicit feature alignment.

Proposed method

• Uses self-training with a mixture of labeled source domain data and proxy-labeled target domain data to iteratively improve model generalization.
• Employs a dual-decoder architecture where prediction agreement between branches generates an agreement map to estimate sample difficulty.
• Applies gradient filtering to prevent updates on low-agreement (hard) samples, focusing training on reliable predictions.
• Implements class-wise loss weighting to address class imbalance in both source and target domains.
• Applies strategic curriculum scheduling: gradually increasing the proportion of target domain data while prioritizing easy proxy-labeled samples and hard source-labeled samples.
• Uses confidence-based mining—easy samples from target domain and hard samples from source domain—to guide the training process.

Experimental results

Research questions

• RQ1 Can a curriculum-based training strategy that combines proxy-labeled target data and hard examples from the source domain improve unsupervised domain adaptation in semantic segmentation?
• RQ2 How does uncertainty estimation via dual-decoder agreement improve model robustness and generalization in low-resource adaptation settings?
• RQ3 Does the proposed method outperform existing feature alignment-based approaches in synthetic-to-real and cross-city domain adaptation tasks?
• RQ4 To what extent can proxy labeling and selective sample mining reduce error amplification during self-training?
• RQ5 Can the method generalize to diverse target domains such as BDD100k, even when source and target domains differ significantly in resolution and content?

Key findings

• The method achieves state-of-the-art mean Intersection over Union (mIoU) of 48.4 on the GTA5 → Cityscapes benchmark, outperforming all prior unsupervised domain adaptation methods using only GTA5 labels.
• On 9 of 19 classes, the method improves IoU over the best existing method, with particularly strong gains on classes like car and person.
• The approach reduces error amplification in self-training by filtering gradients based on prediction agreement, leading to more stable convergence.
• The dual-decoder agreement map effectively highlights ambiguous regions and class boundaries, providing visual interpretability of model uncertainty.
• The method achieves competitive performance on BDD100k, with adaptation from GTA5 yielding slightly higher mIoU than from Cityscapes, indicating strong suitability for synthetic-to-real adaptation.
• The approach is orthogonal to adversarial feature alignment methods like CyCADA, and the two could be combined for further performance gains, though hyperparameter tuning remains challenging.