[Paper Review] Conditional Image-to-Image Translation

Jianxin Lin, Yingce Xia+3 more|arXiv (Cornell University)|May 1, 2018

Generative Adversarial Networks and Image Synthesis22 references19 citations

TL;DR

This paper proposes conditional image-to-image translation using conditional GANs and dual learning to enable fine-grained control over translation results by conditioning on a target-domain image. The method disentangles domain-independent and domain-specific features, allowing diverse, controllable outputs—evidenced by superior results in face-to-face and edges-to-shoes/bags translation tasks with user study validation.

ABSTRACT

Image-to-image translation tasks have been widely investigated with Generative Adversarial Networks (GANs) and dual learning. However, existing models lack the ability to control the translated results in the target domain and their results usually lack of diversity in the sense that a fixed image usually leads to (almost) deterministic translation result. In this paper, we study a new problem, conditional image-to-image translation, which is to translate an image from the source domain to the target domain conditioned on a given image in the target domain. It requires that the generated image should inherit some domain-specific features of the conditional image from the target domain. Therefore, changing the conditional image in the target domain will lead to diverse translation results for a fixed input image from the source domain, and therefore the conditional input image helps to control the translation results. We tackle this problem with unpaired data based on GANs and dual learning. We twist two conditional translation models (one translation from A domain to B domain, and the other one from B domain to A domain) together for inputs combination and reconstruction while preserving domain independent features. We carry out experiments on men's faces from-to women's faces translation and edges to shoes&bags translations. The results demonstrate the effectiveness of our proposed method.

Motivation & Objective

• To address the lack of control and diversity in existing image-to-image translation models, which produce deterministic outputs for fixed inputs.
• To enable fine-grained manipulation of translated image styles by conditioning on a reference image from the target domain.
• To develop a method that preserves domain-independent features from the source image while transferring domain-specific features from the conditional target image.
• To achieve this using unpaired data via GANs and dual learning, ensuring reconstruction and domain consistency.

Proposed method

• The model uses two conditional translation networks: one from domain A to B and the other from B to A, sharing a common feature extraction backbone.
• Domain-independent and domain-specific features are disentangled via separate encoders for source and conditional images.
• Conditional GANs are trained with adversarial loss to ensure generated images are realistic and belong to the target domain.
• Dual learning is applied with reconstruction losses on both domain-independent and domain-specific features to preserve identity and structure.
• Skip connections from domain-specific features to the generator are used to improve reconstruction fidelity and feature consistency.
• The model is trained end-to-end with combined losses: adversarial loss, cycle consistency loss, and feature reconstruction losses for both domain types.

Experimental results

Research questions

• RQ1 Can conditioning on a target-domain image enable diverse and controllable image-to-image translation results for a fixed source input?
• RQ2 How can domain-independent and domain-specific features be effectively disentangled and manipulated in image translation?
• RQ3 Can unpaired data be effectively leveraged to train a conditional translation model without paired examples?
• RQ4 What architectural components (e.g., skip connections, feature reconstruction) are essential for high-quality conditional translation?
• RQ5 How does the proposed method compare to existing GAN-based and dual learning methods in terms of visual quality and user preference?

Key findings

• The proposed cd-GAN model outperforms baseline models (cd-GAN-rec, cd-GAN-nof, etc.) in image quality and feature consistency, especially in preserving domain-independent structure.
• User studies with 17 subjects show that generated images from the proposed model are significantly more similar to the conditional image in domain-specific features than other methods.
• Ablation studies confirm that skip connections and feature reconstruction losses are essential for high-fidelity image reconstruction and disentanglement.
• The model successfully generates images using only domain-specific features, which resemble the conditional image in style, validating effective feature extraction.
• The model also preserves the main shape of the source image when only domain-independent features are used, confirming the disentanglement of structural and style components.
• Results on face-to-face and edges-to-shoes/bags translation tasks demonstrate the method’s effectiveness across symmetric and asymmetric translation settings.