[Paper Review] Domain Stylization: A Strong, Simple Baseline for Synthetic to Real Image Domain Adaptation

Ayşegül Dündar, Ming-Yu Liu+3 more|arXiv (Cornell University)|Jul 24, 2018

Generative Adversarial Networks and Image Synthesis36 references72 citations

TL;DR

The paper proposes Domain Stylization (DS), a simple, non-GAN approach that stylizes synthetic images using randomly paired real images to bridge the synthetic-real gap, achieving state-of-the-art results on semantic segmentation and object detection without training GANs.

ABSTRACT

Deep neural networks have largely failed to effectively utilize synthetic data when applied to real images due to the covariate shift problem. In this paper, we show that by applying a straightforward modification to an existing photorealistic style transfer algorithm, we achieve state-of-the-art synthetic-to-real domain adaptation results. We conduct extensive experimental validations on four synthetic-to-real tasks for semantic segmentation and object detection, and show that our approach exceeds the performance of any current state-of-the-art GAN-based image translation approach as measured by segmentation and object detection metrics. Furthermore we offer a distance based analysis of our method which shows a dramatic reduction in Frechet Inception distance between the source and target domains, offering a quantitative metric that demonstrates the effectiveness of our algorithm in bridging the synthetic-to-real gap.

Motivation & Objective

• Address covariate shift when using synthetic data for real-world vision tasks.
• Propose a simple, non-GAN method to stylize synthetic images using real images while preserving semantic content.
• Demonstrate state-of-the-art performance on semantic segmentation and object detection across multiple benchmarks.
• Provide a Frechet Inception Distance (FID) analysis to quantify distribution alignment between synthetic and real domains.
• Offer practical insights into iterative domain stylization and its effect on performance.

Proposed method

• Use an existing photorealistic style transfer algorithm (FastPhotoStyle) to stylize synthetic images with randomly selected real images.
• Preserve semantic content by keeping segmentation masks associated with synthetic images unchanged during stylization.
• Iteratively train a semantic segmentation network and re-stylize synthetic data using predicted real-image masks to refine style transfer (Algorithm 1).
• Generate stylized datasets D^S by pairing synthetic images with real styles; train SSL networks on stylized data.
• Evaluate DS against GAN-based translation and domain randomization baselines on Cityscapes (GTA/SYNTHIA to Cityscapes) and NYU/ SUNCG (SUNCG to NYU).
• Use Frechet Inception Distance (FID) to quantify domain alignment and show reductions when using segmentation-aware stylization.

Experimental results

Research questions

• RQ1 Does a photorealistic style transfer-based DS method outperform GAN-based image translation for synthetic-to-real domain adaptation?
• RQ2 How does stylizing synthetic images with real styles (with and without segmentation maps) affect segmentation and detection performance?
• RQ3 Can DS reduce distribution shift quantitatively as measured by FID, and how does segmentation mask granularity influence this?
• RQ4 What is the impact of iterative DS and SSL on final performance across tasks (street scenes, indoor scenes, object detection)?

Key findings

• DS achieves superior or competitive accuracy compared to GAN-based translation methods on GTA→Cityscapes and SYNTHIA→Cityscapes for semantic segmentation.
• DS with segmentation maps yields larger improvements than DS without masks in several setups.
• DS improves KITTI object detection performance over synthetic baselines and outperforms several GAN-based methods.
• FID analyses show that DS markedly reduces synthetic-to-real distribution distance, especially when segmentation maps are used.
• An iterative DS/SSL procedure yields further gains, with diminishing returns after two iterations.