[论文解读] Unsupervised Image-to-Image Translation with Generative Adversarial Networks

Hao Dong, Paarth Neekhara+2 more|arXiv (Cornell University)|Jan 10, 2017

Generative Adversarial Networks and Image Synthesis参考文献 18被引用 70

一句话总结

本文提出一种两步无监督方法，使用条件GAN和图像编码器在域间双向、领域无关的方式翻译图像。

ABSTRACT

It's useful to automatically transform an image from its original form to some synthetic form (style, partial contents, etc.), while keeping the original structure or semantics. We define this requirement as the "image-to-image translation" problem, and propose a general approach to achieve it, based on deep convolutional and conditional generative adversarial networks (GANs), which has gained a phenomenal success to learn mapping images from noise input since 2014. In this work, we develop a two step (unsupervised) learning method to translate images between different domains by using unlabeled images without specifying any correspondence between them, so that to avoid the cost of acquiring labeled data. Compared with prior works, we demonstrated the capacity of generality in our model, by which variance of translations can be conduct by a single type of model. Such capability is desirable in applications like bidirectional translation

研究动机与目标

• Aim to translate images between domains without paired data, preserving semantics and structure.
• Learn a universal mapping that can handle multiple domain translations with a single model.
• Leverage shared latent features to enable bidirectional translation between domains.

提出的方法

• Use auxiliary classifier GAN to learn global shared features across domains, representing them as a latent z in [-1, 1].
• Train a conditional generator that produces target-domain images conditioned on domain labels and latent z.
• Introduce an image encoder E that maps real images to the latent space z by enforcing reconstruction of z through the generator, via MSE loss.
• Perform translation by mapping an input image to z and then generating the target-domain image conditioned on the desired label.
• Training is conducted in two steps: step 1 train G across all domains; step 2 train E across all domains with the generator fixed.

实验结果

研究问题

• RQ1 Can unsupervised learning with a shared latent representation enable bidirectional translation across multiple image domains?
• RQ2 Does a two-step framework (generator learning followed by encoder learning) improve reconstruction and translation quality compared to end-to-end approaches?
• RQ3 Can a universal learning approach handle diverse translation tasks (e.g., gender, facial attributes) with a single model?

主要发现

• Demonstrated bidirectional translation on CelebA for gender transformation and on presidential debate videos for face swapping.
• The method learns to translate while preserving background and expressions, indicating effective semantic preservation.
• The two-step training leverages synthetic data from the trained generator to train the image encoder, increasing data efficiency and representation capacity.
• Translations are achieved by applying the encoder to obtain z and then using the conditional generator to synthesize the target-domain image.
• The approach supports learning across multiple domains with a single model, highlighting the universality of the learning algorithm.