[Paper Review] Multi-Target Domain Adaptation with Collaborative Consistency Learning

Takashi Isobe, Xu Jia+6 more|arXiv (Cornell University)|Jun 7, 2021

Domain Adaptation and Few-Shot Learning41 references13 citations

TL;DR

This paper proposes a collaborative consistency learning framework for multi-target domain adaptation in semantic segmentation, where multiple domain-specific expert models are trained with cross-domain consistency regularization and jointly distill knowledge into a single student model. The method achieves state-of-the-art performance across multiple target domains, outperforming both individual domain-adapted models and prior multi-target adaptation approaches on GTA5-to-Cityscapes, IDD, and Mapillary benchmarks.

ABSTRACT

Recently unsupervised domain adaptation for the semantic segmentation task has become more and more popular due to high-cost of pixel-level annotation on real-world images. However, most domain adaptation methods are only restricted to single-source-single-target pair, and can not be directly extended to multiple target domains. In this work, we propose a collaborative learning framework to achieve unsupervised multi-target domain adaptation. An unsupervised domain adaptation expert model is first trained for each source-target pair and is further encouraged to collaborate with each other through a bridge built between different target domains. These expert models are further improved by adding the regularization of making the consistent pixel-wise prediction for each sample with the same structured context. To obtain a single model that works across multiple target domains, we propose to simultaneously learn a student model which is trained to not only imitate the output of each expert on the corresponding target domain, but also to pull different expert close to each other with regularization on their weights. Extensive experiments demonstrate that the proposed method can effectively exploit rich structured information contained in both labeled source domain and multiple unlabeled target domains. Not only does it perform well across multiple target domains but also performs favorably against state-of-the-art unsupervised domain adaptation methods specially trained on a single source-target pair

Motivation & Objective

• To address the challenge of unsupervised multi-target domain adaptation (MTDA) in semantic segmentation, where a single model must generalize across multiple unlabeled target domains.
• To overcome the limitations of single-source-single-target domain adaptation, which cannot be directly extended to multiple target domains.
• To improve generalization and performance by enabling knowledge sharing among domain-specific expert models through collaborative consistency learning.
• To develop a single, robust student model that performs well across all target domains by leveraging online knowledge distillation from multiple experts.
• To explore the benefits of inter-domain connections—both between source-target pairs and among target domains—to enhance transferability and model robustness.

Proposed method

• For each target domain, a domain-specific expert model is trained using adversarial domain adaptation and a consistency loss that aligns predictions of the same content under different domain styles.
• A style translation bridge is constructed to transform source and target images into the style of each target domain, enabling cross-domain consistency learning between different target domains.
• Collaborative consistency learning enforces agreement among expert models by regularizing them to produce consistent pixel-wise predictions for images with the same semantic content but different domain styles.
• A student model is trained via online knowledge distillation to mimic the outputs of all expert models on their respective target domains, while also being regularized to align with the weights of the experts.
• The method incorporates a weight regularization loss to encourage the student model to maintain similarity to the expert models during training, improving generalization.
• The framework is trained end-to-end, allowing the student to benefit from both the specialized knowledge of individual experts and the shared invariance learned through cross-domain consistency.

Experimental results

Research questions

• RQ1 Can a single model be effectively trained to generalize across multiple unlabeled target domains in semantic segmentation without requiring per-domain fine-tuning?
• RQ2 How can knowledge from multiple domain-specific expert models be effectively combined to improve performance on all target domains simultaneously?
• RQ3 Does enforcing consistency in predictions across different domain styles (via a style bridge) lead to better generalization than standard domain adaptation?
• RQ4 Can online knowledge distillation from multiple experts improve the performance of a single student model compared to training individual models per target domain?
• RQ5 To what extent does collaborative learning among experts, including cross-domain consistency, outperform naive ensemble or combined training strategies?

Key findings

• The proposed method achieves 46.7% mIoU on Cityscapes, 47.0% on IDD, and 49.9% on Mapillary when adapting from GTA5, outperforming individual STDA baselines that achieve 43.3%, 43.6%, and 45.8% mIoU respectively.
• On the real-to-real MTDA setting, the method achieves 58.5% mIoU on Mapillary when using Cityscapes as source, surpassing the baseline and prior methods.
• Ablation studies show that combining online knowledge distillation and weight regularization improves mIoU by 1.7% on Cityscapes and 1.8% on IDD compared to the baseline.
• The collaborative consistency learning component alone improves performance by 1.0% on Cityscapes and 1.3% on IDD over the baseline model.
• The method consistently outperforms both individual expert models and prior MTDA methods on all target domains, demonstrating superior generalization and robustness.
• The framework achieves state-of-the-art performance on multi-target adaptation benchmarks, including both synthetic-to-real and real-to-real settings.