[Paper Review] Image-to-image translation for cross-domain disentanglement

Abel González-García, Joost van de Weijer+1 more|arXiv (Cornell University)|May 24, 2018

Generative Adversarial Networks and Image SynthesisComputer Science44 references87 citations

TL;DR

The paper introduces cross-domain disentanglement to separate shared and domain-exclusive factors in image representations, enabling bidirectional multi-modal image translation and cross-domain retrieval without labels, with demonstrated improvements over state-of-the-art baselines on challenging datasets.

ABSTRACT

Deep image translation methods have recently shown excellent results, outputting high-quality images covering multiple modes of the data distribution. There has also been increased interest in disentangling the internal representations learned by deep methods to further improve their performance and achieve a finer control. In this paper, we bridge these two objectives and introduce the concept of cross-domain disentanglement. We aim to separate the internal representation into three parts. The shared part contains information for both domains. The exclusive parts, on the other hand, contain only factors of variation that are particular to each domain. We achieve this through bidirectional image translation based on Generative Adversarial Networks and cross-domain autoencoders, a novel network component. Our model offers multiple advantages. We can output diverse samples covering multiple modes of the distributions of both domains, perform domain-specific image transfer and interpolation, and cross-domain retrieval without the need of labeled data, only paired images. We compare our model to the state-of-the-art in multi-modal image translation and achieve better results for translation on challenging datasets as well as for cross-domain retrieval on realistic datasets.

Motivation & Objective

• Motivate disentangling latent representations to separate domain-shared and domain-exclusive factors.
• Develop a bidirectional image translation framework that enforces three-part representations (shared, X-exclusive, Y-exclusive).
• Enable diverse, multi-modal translations and cross-domain retrieval through a cross-domain autoencoder and GRL-based constraints.
• Show that the disentangled representation supports domain-specific transfer, interpolation, and visual analogies without requiring labeled data.
• Evaluate on MNIST variations and more complex datasets to demonstrate advantages over state-of-the-art multi-modal translation methods.

Proposed method

• Partition latent codes into shared S and exclusive E parts for each domain (E^X, E^Y) and a shared component S^{X} / S^{Y} that is domain-invariant.
• Use bidirectional GAN-based translators G and F with encoders/decoders that operate on (S,E) but generate images using only S and random noise z for the exclusive part.
• Apply a Gradient Reversal Layer (GRL) to a small decoder attempting to reconstruct the target domain from exclusive features to discourage domain-specific leakage.
• Enforce similarity of shared representations across domains with an L1 loss plus controlled noise to avoid collapse (S^X ≈ S^Y, with noise injected to prevent trivial solutions).
• Introduce cross-domain autoencoders to align latent spaces by reconstructing inputs from a mix of same-domain shared and cross-domain exclusive components.
• Train with a WGAN-GP loss for stability and a reconstruction-based autoencoder loss to preserve cross-domain consistency across translations.

Experimental results

Research questions

• RQ1 Can a three-part latent representation (shared, X-exclusive, Y-exclusive) disentangle domain-variant factors while enabling cross-domain translation?
• RQ2 Does cross-domain autoencoding and GRL-based constraints improve multi-modal translation and cross-domain retrieval without labeled data?
• RQ3 How does the proposed representation support sample diversity, domain-specific transfer, and interpolation across two domains?
• RQ4 Can the model achieve effective cross-domain retrieval and visual analogies using only paired data?
• RQ5 Do ablations show necessity of cross-domain autoencoders, GRL, and L1 agreement on shared features?

Key findings

• The model yields diverse multi-modal translations by varying the exclusive noise z while keeping the shared part fixed.
• Cross-domain retrieval using shared features achieves high Recall@1, outperforming pixel-based baselines and enabling label-free learning.
• Visual analogies and domain-specific interpolations are achievable by manipulating shared and exclusive components, often without explicit supervision.
• Ablation results indicate cross-domain autoencoders and GRL contribute significantly to disentanglement and translation quality; removing them degrades performance.
• The method scales to realistic datasets (maps/satellite, facades/maps) and achieves competitive LPIPS scores on car and chair datasets for bi-directional translation.
• Compared to BicycleGAN and pix2pix baselines, the proposed approach yields better perceptual quality and mode diversity, particularly when handling significant structural changes.