[Paper Review] Phase Consistent Ecological Domain Adaptation

Yanchao Yang, Dong Lao+2 more|arXiv (Cornell University)|Apr 10, 2020

Domain Adaptation and Few-Shot Learning45 references4 citations

TL;DR

This paper proposes Phase Consistent Ecological Domain Adaptation (PCEDA), a novel unsupervised domain adaptation framework for semantic segmentation that enforces phase consistency in image-to-image translation and leverages ecological statistics via a Conditional Prior Network (CPN). By integrating these two priors into a deep learning pipeline, PCEDA achieves state-of-the-art performance on GTA5-to-Cityscapes and Synthia-to-Cityscapes benchmarks, with improvements of up to 8.0% over prior methods.

ABSTRACT

We introduce two criteria to regularize the optimization involved in learning a classifier in a domain where no annotated data are available, leveraging annotated data in a different domain, a problem known as unsupervised domain adaptation. We focus on the task of semantic segmentation, where annotated synthetic data are aplenty, but annotating real data is laborious. The first criterion, inspired by visual psychophysics, is that the map between the two image domains be phase-preserving. This restricts the set of possible learned maps, while enabling enough flexibility to transfer semantic information. The second criterion aims to leverage ecological statistics, or regularities in the scene which are manifest in any image of it, regardless of the characteristics of the illuminant or the imaging sensor. It is implemented using a deep neural network that scores the likelihood of each possible segmentation given a single un-annotated image. Incorporating these two priors in a standard domain adaptation framework improves performance across the board in the most common unsupervised domain adaptation benchmarks for semantic segmentation.

Motivation & Objective

• To address the challenge of semantic segmentation in real-world images where dense annotations are expensive, by leveraging abundant synthetic, annotated data.
• To improve unsupervised domain adaptation (UDA) for semantic segmentation by incorporating domain-invariant priors that are not based solely on feature or image distribution alignment.
• To ensure semantic consistency during domain translation by enforcing phase preservation in the Fourier domain, informed by visual psychophysics.
• To model scene-level regularities (ecological statistics) in unannotated target images to guide plausible segmentation hypotheses.
• To integrate both priors into an end-to-end differentiable framework that enhances generalization across domain shift in semantic segmentation.

Proposed method

• Introduces a phase-consistent image translation module that preserves the phase component of the Fourier transform during domain translation, ensuring semantic content is retained.
• Employs a Conditional Prior Network (CPN) to model scene compatibility and score the likelihood of segmentations based on ecological statistics of natural scenes.
• Uses the CPN as a differentiable prior during training to regularize segmentation predictions in the target domain, even without ground-truth annotations.
• Combines phase consistency and ecological prior regularization within a unified optimization framework, enhancing feature and output space alignment.
• Applies the method end-to-end using standard segmentation backbones (e.g., DeepLab-V2 with ResNet-101, VGG-16, DRN-26) under standard UDA settings.
• Incorporates the CPN during training with a 1.5-second overhead per batch, but no inference cost, enabling efficient deployment.

Experimental results

Research questions

• RQ1 Can enforcing phase consistency in image-to-image translation improve semantic segmentation performance in unsupervised domain adaptation?
• RQ2 Can ecological statistics—such as spatial regularity and shape plausibility—be effectively modeled as a data-dependent prior to guide segmentation in the absence of annotations?
• RQ3 How do phase consistency and ecological priors compare in isolation and in combination in improving domain adaptation for semantic segmentation?
• RQ4 Can these priors generalize across different backbone architectures and domain shift scenarios (e.g., GTA5→Cityscapes, Synthia→Cityscapes)?
• RQ5 What is the computational cost of integrating these priors into a standard UDA pipeline?

Key findings

• PCEDA achieves state-of-the-art performance on the GTA5-to-Cityscapes benchmark, with a 4.1% mIoU improvement over the second-best method using ResNet-101.
• On the Synthia-to-Cityscapes benchmark, PCEDA achieves a 4.3% mIoU improvement over the second-best method using ResNet-101, and a 5.4% improvement with VGG-16.
• The ablation study confirms that both phase consistency and the CPN prior contribute significantly to performance, with the full model outperforming each component alone.
• Qualitative results show that PCEDA produces more spatially regularized and scene-consistent segmentations compared to SOTA methods.
• The computational overhead of enforcing phase consistency is negligible (<0.001s per 1024×512 image), and the CPN adds only 1.5s per batch during training, with no runtime cost at inference.
• The method consistently improves mIoU and fwIoU across all evaluated semantic categories, indicating robustness to class-specific domain shifts.