[Paper Review] Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training

Yang Zou, Zhiding Yu+2 more|arXiv (Cornell University)|Oct 18, 2018

Domain Adaptation and Few-Shot Learning43 references50 citations

TL;DR

This paper proposes a unsupervised domain adaptation framework for semantic segmentation using iterative self-training. It introduces class-balanced self-training (CBST) and CBST with spatial priors (CBST-SP) to address class imbalance and spatial structure, achieving state-of-the-art results on synthetic-to-real and cross-city domain shifts.

ABSTRACT

Recent deep networks achieved state of the art performance on a variety of semantic segmentation tasks. Despite such progress, these models often face challenges in real world `wild tasks' where large difference between labeled training/source data and unseen test/target data exists. In particular, such difference is often referred to as `domain gap', and could cause significantly decreased performance which cannot be easily remedied by further increasing the representation power. Unsupervised domain adaptation (UDA) seeks to overcome such problem without target domain labels. In this paper, we propose a novel UDA framework based on an iterative self-training procedure, where the problem is formulated as latent variable loss minimization, and can be solved by alternatively generating pseudo labels on target data and re-training the model with these labels. On top of self-training, we also propose a novel class-balanced self-training framework to avoid the gradual dominance of large classes on pseudo-label generation, and introduce spatial priors to refine generated labels. Comprehensive experiments show that the proposed methods achieve state of the art semantic segmentation performance under multiple major UDA settings.

Motivation & Objective

• Motivate unsupervised domain adaptation for dense semantic segmentation in real-world, domain-shifted settings.
• Develop a self-training framework that treats target labels as latent pseudo-labels and optimizes them jointly with modelparameters.
• Address class-imbalance in pseudo-labels by introducing class-balanced self-training (CBST).
• Enhance pseudo-label quality with spatial priors when source/target domains share layout.

Proposed method

• Frame pseudo-label estimation and network training as a unified loss minimization problem in a deep self-training loop.
• Introduce a self-paced learning regime to progressively select the most confident pseudo-labels (ST).
• Propose CBST by normalizing class-wise confidences to balance pseudo-label generation across classes.
• Derive a CBST variant with spatial priors (CBST-SP) by incorporating spatial class priors qn(c) into pseudo-label selection.
• Provide a practical algorithm for determining the pacing parameter k (and class weights kc) to control pseudo-label inclusion per round.
• Optionally combine CBST-SP with multi-scale testing (MST) for further gains.

Experimental results

Research questions

• RQ1 Can self-training without adversarial training reach competitive/leading performance for unsupervised domain adaptation in semantic segmentation?
• RQ2 Does balancing pseudo-labels across classes (CBST) improve transfer for hard-to-transfer classes compared to vanilla self-training?
• RQ3 Do spatial priors help when source/target domains share similar scene layouts (CBST-SP)?
• RQ4 How do ST, CBST, CBST-SP perform on synthetic-to-real (GTA5/SYNTHIA→Cityscapes) and cross-city (Cityscapes→NTHU) domain shifts?

Key findings

• CBST and CBST-SP achieve state-of-the-art or competitive performance across major UDA settings, including synthetic-to-real and cross-city shifts.
• In GTA5→Cityscapes with ResNet-38, CBST yields 46.2% mean IoU, and with MST (multi-scale testing) the mean IoU reaches 47.0%.
• CBST improves over vanilla ST by addressing per-class transfer difficulty, particularly for initially hard-to-transfer classes.
• CBST-SP further improves adaptation when source and target domains share spatial layouts, outperforming baseline methods in the reported settings.
• In SYNTHIA→Cityscapes, CBST with ResNet-38 achieves superior performance, outperforming ST by a notable margin (e.g., +1.7 mIoU over the best prior method).