[Paper Review] Unsupervised Image-to-Image Translation Using Domain-Specific Variational Information Bound

Hadi Kazemi, Sobhan Soleymani+3 more|arXiv (Cornell University)|Nov 29, 2018

Image Processing Techniques and ApplicationsEngineering20 citations

TL;DR

This paper proposes an unsupervised image-to-image translation framework that disentangles domain-invariant and domain-specific representations using a domain-specific variational information bound. By learning separate codes for shared content and modality-specific attributes, the model enables one-to-many translation without paired data, achieving diverse, realistic outputs via sampling or reference-guided generation.

ABSTRACT

Unsupervised image-to-image translation is a class of computer vision problems which aims at modeling conditional distribution of images in the target domain, given a set of unpaired images in the source and target domains. An image in the source domain might have multiple representations in the target domain. Therefore, ambiguity in modeling of the conditional distribution arises, specially when the images in the source and target domains come from different modalities. Current approaches mostly rely on simplifying assumptions to map both domains into a shared-latent space. Consequently, they are only able to model the domain-invariant information between the two modalities. These approaches usually fail to model domain-specific information which has no representation in the target domain. In this work, we propose an unsupervised image-to-image translation framework which maximizes a domain-specific variational information bound and learns the target domain-invariant representation of the two domain. The proposed framework makes it possible to map a single source image into multiple images in the target domain, utilizing several target domain-specific codes sampled randomly from the prior distribution, or extracted from reference images.

Motivation & Objective

• To address the limitation of shared-latent space models in unsupervised image-to-image translation, which fail to capture modality-specific information when domains differ significantly.
• To enable one-to-many translation by learning disentangled representations of domain-invariant content and domain-specific attributes.
• To improve diversity and realism in unsupervised translation by maximizing domain-specific variational information bounds.
• To overcome cycle-consistency-induced encoding of domain-specific features in the translated image, which degrades performance and stability.

Proposed method

• The framework uses two encoders to extract a domain-invariant code and a domain-specific code from the source image.
• A generator synthesizes target-domain images using the domain-invariant code and a target-domain-specific code sampled from a unit normal prior or extracted from reference images.
• Mutual information maximization between the source domain-specific code and the source image ensures meaningful disentanglement of domain-specific features.
• Mutual information minimization between the source domain-specific code and the translated target image prevents domain-specific features from being encoded in the target, reducing cycle-consistency artifacts.
• The model enforces cycle consistency via reconstruction paths for both domain-invariant and domain-specific codes, ensuring structural and semantic fidelity.
• The loss function includes a domain-specific variational information bound that encourages the model to learn interpretable, disentangled representations of modality-specific content.

Experimental results

Research questions

• RQ1 Can a disentangled representation of domain-invariant and domain-specific features improve one-to-many image translation in an unsupervised setting?
• RQ2 How does maximizing domain-specific variational information bound enhance diversity and realism in image translation without paired data?
• RQ3 To what extent does separating domain-specific and domain-invariant codes reduce the encoding of source-specific attributes in the translated image?
• RQ4 Does the proposed method outperform cycle-consistent models like CycleGAN and UNIT in cross-modality translation tasks with significant domain differences?
• RQ5 Can sampling from a prior or using reference images effectively control the diversity of generated outputs in the target domain?

Key findings

• The proposed framework achieves LPIPS scores of 0.121 (shoes) and 0.129 (handbags) with domain-specific code cycle consistency, significantly outperforming the ablated version (0.095 and 0.113) in diversity.
• The model achieves a lower FID score than CycleGAN and UNIT, indicating better distributional alignment with real data, confirming improved image quality and realism.
• Without domain-specific code cycle consistency, the model suffers from mode collapse, as shown by a drop in LPIPS, indicating reduced output diversity.
• The ablation study confirms that removing domain-invariant code cycle consistency leads to poor image quality, as the invariant code may then carry domain-specific information.
• The framework maintains high fooling rates in human evaluation, indicating that generated images are visually realistic and indistinguishable from real images.
• The model successfully generates diverse, realistic translations across different domains (e.g., edges to shoes, edges to handbags), even without paired data or supervision.