[Paper Review] Unsupervised Image-to-Image Translation with Generative Adversarial Networks

Hao Dong, Paarth Neekhara+2 more|arXiv (Cornell University)|Jan 10, 2017

Generative Adversarial Networks and Image SynthesisComputer Science18 references70 citations

TL;DR

The paper presents a two-step unsupervised method using conditional GANs and an image encoder to translate images across domains in a bidirectional, domain-agnostic way.

ABSTRACT

It's useful to automatically transform an image from its original form to some synthetic form (style, partial contents, etc.), while keeping the original structure or semantics. We define this requirement as the "image-to-image translation" problem, and propose a general approach to achieve it, based on deep convolutional and conditional generative adversarial networks (GANs), which has gained a phenomenal success to learn mapping images from noise input since 2014. In this work, we develop a two step (unsupervised) learning method to translate images between different domains by using unlabeled images without specifying any correspondence between them, so that to avoid the cost of acquiring labeled data. Compared with prior works, we demonstrated the capacity of generality in our model, by which variance of translations can be conduct by a single type of model. Such capability is desirable in applications like bidirectional translation

Motivation & Objective

• Aim to translate images between domains without paired data, preserving semantics and structure.
• Learn a universal mapping that can handle multiple domain translations with a single model.
• Leverage shared latent features to enable bidirectional translation between domains.

Proposed method

• Use auxiliary classifier GAN to learn global shared features across domains, representing them as a latent z in [-1, 1].
• Train a conditional generator that produces target-domain images conditioned on domain labels and latent z.
• Introduce an image encoder E that maps real images to the latent space z by enforcing reconstruction of z through the generator, via MSE loss.
• Perform translation by mapping an input image to z and then generating the target-domain image conditioned on the desired label.
• Training is conducted in two steps: step 1 train G across all domains; step 2 train E across all domains with the generator fixed.

Experimental results

Research questions

• RQ1 Can unsupervised learning with a shared latent representation enable bidirectional translation across multiple image domains?
• RQ2 Does a two-step framework (generator learning followed by encoder learning) improve reconstruction and translation quality compared to end-to-end approaches?
• RQ3 Can a universal learning approach handle diverse translation tasks (e.g., gender, facial attributes) with a single model?

Key findings

• Demonstrated bidirectional translation on CelebA for gender transformation and on presidential debate videos for face swapping.
• The method learns to translate while preserving background and expressions, indicating effective semantic preservation.
• The two-step training leverages synthetic data from the trained generator to train the image encoder, increasing data efficiency and representation capacity.
• Translations are achieved by applying the encoder to obtain z and then using the conditional generator to synthesize the target-domain image.
• The approach supports learning across multiple domains with a single model, highlighting the universality of the learning algorithm.