[Paper Review] Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentation

Yuyang Zhao, Zhun Zhong+3 more|arXiv (Cornell University)|Apr 6, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes SHADE, a style-hallucinated dual consistency learning framework for synthetic-to-real domain generalization in semantic segmentation. By introducing Style Consistency (SC) and Retrospection Consistency (RC) with a novel Style Hallucination Module (SHM), SHADE generates diverse, realistic style-augmented synthetic samples and enforces consistency across them and with real-world knowledge, achieving state-of-the-art performance with a 5.05% and 8.35% mIoU gain on average across three real-world datasets in single- and multi-source settings, respectively.

ABSTRACT

In this paper, we study the task of synthetic-to-real domain generalized semantic segmentation, which aims to learn a model that is robust to unseen real-world scenes using only synthetic data. The large domain shift between synthetic and real-world data, including the limited source environmental variations and the large distribution gap between synthetic and real-world data, significantly hinders the model performance on unseen real-world scenes. In this work, we propose the Style-HAllucinated Dual consistEncy learning (SHADE) framework to handle such domain shift. Specifically, SHADE is constructed based on two consistency constraints, Style Consistency (SC) and Retrospection Consistency (RC). SC enriches the source situations and encourages the model to learn consistent representation across style-diversified samples. RC leverages real-world knowledge to prevent the model from overfitting to synthetic data and thus largely keeps the representation consistent between the synthetic and real-world models. Furthermore, we present a novel style hallucination module (SHM) to generate style-diversified samples that are essential to consistency learning. SHM selects basis styles from the source distribution, enabling the model to dynamically generate diverse and realistic samples during training. Experiments show that our SHADE yields significant improvement and outperforms state-of-the-art methods by 5.05% and 8.35% on the average mIoU of three real-world datasets on single- and multi-source settings, respectively.

Motivation & Objective

• To address the significant domain shift between synthetic and real-world data in semantic segmentation, especially when real-world data are unavailable during training.
• To improve model generalization to unseen real-world scenes by learning domain-invariant representations without relying on extra real-world data.
• To overcome limitations of existing methods that either overfit to synthetic data or depend on real-world data for style transfer.
• To develop a method that explicitly learns consistent representations across diverse styles and aligns with real-world knowledge through retrospective guidance.
• To enable robust performance in both single-source and multi-source domain generalization settings for real-world deployment.

Proposed method

• The framework employs two consistency constraints: Style Consistency (SC), which enforces consistent predictions across style-augmented synthetic samples, and Retrospection Consistency (RC), which aligns the model’s features with those of a pre-trained ImageNet model to reflect real-world knowledge.
• A novel Style Hallucination Module (SHM) generates diverse, realistic style-augmented samples by manipulating channel-wise mean and standard deviation of feature maps, preserving spatial alignment.
• SHM uses Farthest Point Sampling (FPS) to select basis styles from the source distribution, ensuring coverage of rare and diverse styles, including those close to real-world domains.
• The SHM is inserted at early network layers (e.g., after layer0 of ResNet) to maximize style representation while avoiding semantic interference from deep layers.
• The method dynamically reselects basis styles every k=3 epochs to adapt to evolving style distributions during training.
• The dual consistency learning is optimized end-to-end, with SC promoting invariance to style variations and RC preventing overfitting to synthetic data.

Experimental results

Research questions

• RQ1 Can style-augmented synthetic data improve generalization to unseen real-world scenes in domain generalization for semantic segmentation?
• RQ2 How can a model learn domain-invariant representations without access to real-world target data during training?
• RQ3 What is the impact of different style generation strategies on model generalization and robustness?
• RQ4 How does retrospective knowledge from a pre-trained ImageNet model improve generalization to real-world distributions?
• RQ5 Can a style hallucination module that selects diverse basis styles significantly enhance consistency learning and model performance?

Key findings

• SHADE achieves a 5.05% average mIoU improvement over state-of-the-art methods on three real-world datasets in the single-source domain generalization setting.
• In the multi-source setting, SHADE achieves an 8.35% mIoU gain over SOTA methods, demonstrating strong generalization across diverse source domains.
• The ablation study confirms that FPS-based basis selection in SHM outperforms random and K-means-based methods by better covering rare and diverse styles.
• Inserting SHM after the first ResNet block (L0) yields the best performance, as shallow layers preserve more pure style information.
• Re-selecting basis styles every 3 epochs (k=3) maintains optimal style diversity and performance, while single initial selection leads to a drop in mIoU below 41%.
• When trained on CityScapes and generalized to both real (BDD100K, Mapillary) and synthetic (GTAV, SYNTHIA) domains, SHADE outperforms ISW and IBN-Net across all benchmarks.