[Paper Review] CyCADA: Cycle-Consistent Adversarial Domain Adaptation

Judy Hoffman, Eric Tzeng+6 more|arXiv (Cornell University)|Nov 8, 2017

Domain Adaptation and Few-Shot Learning35 references630 citations

TL;DR

CyCADA integrates cycle-consistent image translation with adversarial domain adaptation at pixel and feature levels, plus semantic consistency, to unsupervisedly adapt models across domains, achieving state-of-the-art results in digit recognition and semantic segmentation.

ABSTRACT

Domain adaptation is critical for success in new, unseen environments. Adversarial adaptation models applied in feature spaces discover domain invariant representations, but are difficult to visualize and sometimes fail to capture pixel-level and low-level domain shifts. Recent work has shown that generative adversarial networks combined with cycle-consistency constraints are surprisingly effective at mapping images between domains, even without the use of aligned image pairs. We propose a novel discriminatively-trained Cycle-Consistent Adversarial Domain Adaptation model. CyCADA adapts representations at both the pixel-level and feature-level, enforces cycle-consistency while leveraging a task loss, and does not require aligned pairs. Our model can be applied in a variety of visual recognition and prediction settings. We show new state-of-the-art results across multiple adaptation tasks, including digit classification and semantic segmentation of road scenes demonstrating transfer from synthetic to real world domains.

Motivation & Objective

• Motivate and address the domain shift between synthetic/real or different visual domains in deep models.
• Develop an unsupervised adaptation framework that preserves semantic content during domain translation.
• Unify pixel-space and feature-space adaptation with cycle-consistency and semantic losses.
• Demonstrate state-of-the-art performance on digit classification and urban semantic segmentation tasks.
• Show the interpretability benefits of image-space adaptation alongside performance gains.

Proposed method

• Introduce G_S→T and G_T→S generators with adversarial discriminators to map between source and target domains.
• Impose cycle-consistency loss to ensure mappings are invertible and preserve content.
• Apply semantic consistency loss by using a fixed source task model to enforce invariant labels after translation.
• Optionally include a feature-level GAN loss to align intermediate representations.
• Optimize a joint CyCADA objective that combines task loss, GAN losses, cycle loss, and semantic loss.
• Implement as pixel-to-pixel convnets for G, and FCN/convnets for f and D.

Experimental results

Research questions

• RQ1 Can cycle-consistent adversarial mappings preserve semantic content while translating across domains?
• RQ2 Does multi-level adaptation (pixel and feature) improve unsupervised domain adaptation over single-level methods?
• RQ3 How do cycle-consistency and semantic consistency influence performance on digit recognition and semantic segmentation under domain shift?
• RQ4 What is the impact of image-space (pixel) adaptation in addition to feature-space adaptation on interpretability and performance?
• RQ5 To what extent can CyCADA close the gap between source-trained and target-trained performance in synthetic-to-real scenarios?

Key findings

• CyCADA achieves state-of-the-art results on digit adaptation across USPS, MNIST, and SVHN shifts.
• Pixel-space adaptation alone yields strong results for small domain shifts, with added feature-space adaptation providing further gains for larger shifts.
• In semantic segmentation, CyCADA attains state-of-the-art performance in synthetic-to-real tasks and cross-season adaptation, approaching target-supervised oracle levels on several metrics.
• Joint pixel-space and feature-space adaptation yields the best overall performance in digit classification tasks.
• Image-space adaptation provides interpretable visual translations (e.g., winter from fall in SYNTHIA) that correlate with improved segmentation performance.