[Paper Review] Semantic-aware Grad-GAN for Virtual-to-Real Urban Scene Adaption

Peilun Li, Xiaodan Liang+2 more|arXiv (Cornell University)|Jan 5, 2018

Advanced Vision and ImagingComputer Science41 references58 citations

TL;DR

Introduces SG-GAN, a semantic-aware, gradient-guided GAN for virtual-to-real urban scene adaptation that preserves semantic boundaries and uses a semantic-aware discriminator to render region-specific textures, improving downstream semantic segmentation.

ABSTRACT

Recent advances in vision tasks (e.g., segmentation) highly depend on the availability of large-scale real-world image annotations obtained by cumbersome human labors. Moreover, the perception performance often drops significantly for new scenarios, due to the poor generalization capability of models trained on limited and biased annotations. In this work, we resort to transfer knowledge from automatically rendered scene annotations in virtual-world to facilitate real-world visual tasks. Although virtual-world annotations can be ideally diverse and unlimited, the discrepant data distributions between virtual and real-world make it challenging for knowledge transferring. We thus propose a novel Semantic-aware Grad-GAN (SG-GAN) to perform virtual-to-real domain adaption with the ability of retaining vital semantic information. Beyond the simple holistic color/texture transformation achieved by prior works, SG-GAN successfully personalizes the appearance adaption for each semantic region in order to preserve their key characteristic for better recognition. It presents two main contributions to traditional GANs: 1) a soft gradient-sensitive objective for keeping semantic boundaries; 2) a semantic-aware discriminator for validating the fidelity of personalized adaptions with respect to each semantic region. Qualitative and quantitative experiments demonstrate the superiority of our SG-GAN in scene adaption over state-of-the-art GANs. Further evaluations on semantic segmentation on Cityscapes show using adapted virtual images by SG-GAN dramatically improves segmentation performance than original virtual data. We release our code at https://github.com/Peilun-Li/SG-GAN.

Motivation & Objective

• Motivate reducing annotation burden by transferring knowledge from automatically generated virtual-world annotations to real-world perception tasks.
• Preserve semantic information while adapting appearance across semantic regions (e.g., road, car) to closer real-world distributions.
• Introduce a soft gradient-sensitive objective to maintain semantic boundaries during adaptation.
• Introduce a semantic-aware discriminator that evaluates fidelity per semantic region to avoid uniform global changes.

Proposed method

• Propose SG-GAN with two generators for virtual-to-real and real-to-virtual adaptation and two semantic-aware discriminators.
• Incorporate adversarial loss, cycle-consistency loss, and a novel soft gradient-sensitive loss that aligns image gradients with semantic boundaries.
• Define L_grad to focus on boundary-consistent gradient differences, enabling region-specific texture changes.
• Develop SD_V and SD_R to enforce semantic-region-aware realism, using one-hot semantic masks to modulate discriminator features.
• Train on unpaired GTA-V and Cityscapes data, with variants SG-GAN-2K and SG-GAN-25K to study data scale effects.
• Evaluate on Cityscapes segmentation to show improved performance using adapted virtual data.

Experimental results

Research questions

• RQ1 Can virtual-world labeled data be effectively translated to real-world distributions without losing semantic integrity?
• RQ2 Does a gradient-sensitive objective and semantic-aware discriminator improve boundary preservation and region-specific texture rendering during virtual-to-real adaptation?
• RQ3 To what extent does increasing virtual-world training data improve adaptation and downstream segmentation performance?

Key findings

• SG-GAN yields qualitatively sharper semantic boundaries and finer region-specific textures compared to baselines.
• A/B tests show SG-GAN outperforms CycleGAN, DualGAN, SimGAN, and BiGAN in realism judgments.
• Adapting GTA-V data with SG-GAN substantially improves Cityscapes semantic segmentation over using original virtual data.
• Semantic-aware discriminators enable region-specific tone and texture changes (e.g., traffic lights, sky) not captured by global discriminators.
• Increasing virtual data from 2K to 25K provides additional but diminishing gains in realism and segmentation metrics.