[Paper Review] Unsupervised Domain Adaptation for Video Semantic Segmentation

Inkyu Shin, Kwanyong Park+2 more|arXiv (Cornell University)|Jul 23, 2021

Domain Adaptation and Few-Shot Learning32 references4 citations

TL;DR

This paper proposes a novel two-phase unsupervised domain adaptation framework for video semantic segmentation, combining Video Adversarial Training (VAT) with temporal feature alignment and Video Self-Training (VST) with flow-guided pseudo label refinement. It achieves state-of-the-art performance on the VIPER to Cityscapes-VPS benchmark, outperforming all prior image-based UDA methods in both mIoU and VPQ metrics by leveraging spatio-temporal context.

ABSTRACT

Unsupervised Domain Adaptation for semantic segmentation has gained immense popularity since it can transfer knowledge from simulation to real (Sim2Real) by largely cutting out the laborious per pixel labeling efforts at real. In this work, we present a new video extension of this task, namely Unsupervised Domain Adaptation for Video Semantic Segmentation. As it became easy to obtain large-scale video labels through simulation, we believe attempting to maximize Sim2Real knowledge transferability is one of the promising directions for resolving the fundamental data-hungry issue in the video. To tackle this new problem, we present a novel two-phase adaptation scheme. In the first step, we exhaustively distill source domain knowledge using supervised loss functions. Simultaneously, video adversarial training (VAT) is employed to align the features from source to target utilizing video context. In the second step, we apply video self-training (VST), focusing only on the target data. To construct robust pseudo labels, we exploit the temporal information in the video, which has been rarely explored in the previous image-based self-training approaches. We set strong baseline scores on 'VIPER to CityscapeVPS' adaptation scenario. We show that our proposals significantly outperform previous image-based UDA methods both on image-level (mIoU) and video-level (VPQ) evaluation metrics.

Motivation & Objective

• To address the lack of video-specific domain adaptation techniques in unsupervised semantic segmentation.
• To improve Sim2Real transferability by leveraging large-scale synthetic video data with minimal real-world annotation.
• To develop a method that effectively utilizes temporal context across video frames for robust domain adaptation.
• To establish a new benchmark for unsupervised video domain adaptation in complex urban scenes.
• To demonstrate that video-specific adaptation significantly outperforms image-level adaptation in video segmentation tasks.

Proposed method

• Proposes a two-phase framework: Video Adversarial Training (VAT) followed by Video Self-Training (VST).
• In VAT, a sequence discriminator uses spatio-temporal features to align source and target domain distributions.
• Introduces a tube matching loss to explicitly model temporal changes in object boundaries and shapes.
• In VST, pseudo labels are generated using flow-based warping from neighboring frames to improve consistency.
• Employs a cut-out-based refinement strategy that discards inconsistent pseudo labels instead of filling in missing regions.
• Uses a flow-based warping function $ W_{t- au \Rightarrow t} $ to propagate labels from reference frames, combined with a non-occlusion mask $ \mathbb{I}_{t-\tau \Rightarrow t} $.

Experimental results

Research questions

• RQ1 Can a two-phase video domain adaptation framework significantly improve semantic segmentation performance on real-world video data compared to image-based UDA methods?
• RQ2 How does explicit modeling of temporal context enhance domain adaptation in video semantic segmentation?
• RQ3 What is the impact of pseudo label refinement strategies—fill-in vs. cut-out—on final segmentation accuracy?
• RQ4 To what extent does flow-based label propagation improve pseudo label quality in the absence of ground-truth labels?
• RQ5 Is the proposed method robust to domain shift in complex urban driving scenes, such as from simulation to real-world Cityscapes-VPS?

Key findings

• The proposed method achieves 51.91 mIoU and 36.43 VPQ$^s$ on the VIPER to Cityscapes-VPS benchmark, significantly outperforming all prior image-based UDA methods.
• The cut-out-based pseudo label refinement strategy improves performance over the fill-in baseline, reducing error accumulation from noisy pseudo labels.
• Pseudo label quality is substantially enhanced by temporal aggregation, as shown by a P-mIoU of 68.8 at 30% label proportion, surpassing class-balanced and instance-adaptive baselines.
• Qualitative results demonstrate consistent and clear predictions across video frames, confirming the method's robustness and temporal coherence.
• The method shows superior generalization on complex urban scenes, with consistent performance across diverse object categories and motion patterns.
• Limitations include dependency on FlowNet for warping, which can degrade performance on classes like 'sidewalk' due to motion estimation errors.