[Paper Review] DRIT++: Diverse Image-to-Image Translation via Disentangled Representations

Hsin-Ying Lee, Hung-Yu Tseng+5 more|arXiv (Cornell University)|May 2, 2019

Generative Adversarial Networks and Image Synthesis64 references185 citations

TL;DR

DRIT++ learns multimodal, unpaired image-to-image translation by disentangling content (domain-invariant) and attribute (domain-specific) representations, enabling diverse, realistic outputs and multi-domain translation.

ABSTRACT

Image-to-image translation aims to learn the mapping between two visual domains. There are two main challenges for this task: 1) lack of aligned training pairs and 2) multiple possible outputs from a single input image. In this work, we present an approach based on disentangled representation for generating diverse outputs without paired training images. To synthesize diverse outputs, we propose to embed images onto two spaces: a domain-invariant content space capturing shared information across domains and a domain-specific attribute space. Our model takes the encoded content features extracted from a given input and attribute vectors sampled from the attribute space to synthesize diverse outputs at test time. To handle unpaired training data, we introduce a cross-cycle consistency loss based on disentangled representations. Qualitative results show that our model can generate diverse and realistic images on a wide range of tasks without paired training data. For quantitative evaluations, we measure realism with user study and Fréchet inception distance, and measure diversity with the perceptual distance metric, Jensen-Shannon divergence, and number of statistically-different bins.

Motivation & Objective

• Address lack of paired training data in image-to-image translation.
• Achieve multimodal, diverse outputs from a single input without supervision.
• Disentangle representations into domain-invariant content and domain-specific attributes.
• Extend to multi-domain image-to-image translation.
• Improve diversity without sacrificing realism using regularization and cross-cycle constraints.

Proposed method

• Embed images into a shared content space and domain-specific attribute spaces using encoders.
• Use a content discriminator and weight-sharing to enforce a shared content space across domains.
• Apply a cross-cycle consistency loss by swapping attribute representations across domains and reconstructing inputs.
• Incorporate mode-seeking regularization to enhance output diversity.
• Extend the framework to multi-domain translation with a single generator and domain classifier.
• Optionally perform example-guided attribute transfer using learned attribute vectors.

Experimental results

Research questions

• RQ1 Can diverse, realistic I2I translations be learned from unpaired data via disentangled representations?
• RQ2 Does separating content and attributes enable multimodal outputs and attribute transfer across and within domains?
• RQ3 Can the approach scale to multi-domain I2I translation with a single generator?
• RQ4 What effects do the content discriminator and mode-seeking regularization have on realism and diversity?

Key findings

• DRIT++ achieves diverse and realistic translations on multiple unpaired I2I tasks.
• Cross-cycle consistency with disentangled representations enables reliable reconstruction from non-corresponding image pairs.
• Mode-seeking regularization significantly improves diversity and mitigates mode collapse.
• Content discriminator reduces domain-specific leakage in the shared content space, aligning domain representations.
• Multi-domain translation with a single generator yields diverse results across several domains (real images and artistic styles, varied weather).
• Quantitative metrics (FID, LPIPS, JSD, NDB) show DRIT++ outperforming several baselines on tested tasks.