[Paper Review] Toward Multimodal Image-to-Image Translation

Jun-Yan Zhu, Richard Zhang+5 more|arXiv (Cornell University)|Nov 30, 2017

Generative Adversarial Networks and Image Synthesis57 references739 citations

TL;DR

The paper introduces BicycleGAN, a family of multimodal conditional image-to-image translation models that enforce a bidirectional link between latent codes and outputs to produce diverse and realistic results, addressing mode collapse in prior GAN-based approaches.

ABSTRACT

Many image-to-image translation problems are ambiguous, as a single input image may correspond to multiple possible outputs. In this work, we aim to model a \emph{distribution} of possible outputs in a conditional generative modeling setting. The ambiguity of the mapping is distilled in a low-dimensional latent vector, which can be randomly sampled at test time. A generator learns to map the given input, combined with this latent code, to the output. We explicitly encourage the connection between output and the latent code to be invertible. This helps prevent a many-to-one mapping from the latent code to the output during training, also known as the problem of mode collapse, and produces more diverse results. We explore several variants of this approach by employing different training objectives, network architectures, and methods of injecting the latent code. Our proposed method encourages bijective consistency between the latent encoding and output modes. We present a systematic comparison of our method and other variants on both perceptual realism and diversity.

Motivation & Objective

• Motivate modeling a distribution of plausible outputs for ambiguous image-to-image translation tasks.
• Develop a low-dimensional latent space to capture output modalities not present in the input.
• Enforce bijective consistency between latent codes and outputs to mitigate mode collapse.
• Explore and compare several training objectives and architectures to maximize realism and diversity.
• Provide a unified framework (BicycleGAN) that combines conditional VAE-GAN and latent regressor approaches.

Proposed method

• Extend pix2pix with a low-dimensional latent code z drawn from a prior p(z) to generate diverse outputs G(A, z).
• Introduce and compare three approaches: cVAE-GAN (B encoded from B with KL regularization), cLR-GAN (latent regressor enforcing z recovery from G(A, z)), and their combination into BicycleGAN.
• Enforce bidirectional constraints: B -> z -> G(A, z) and z -> G(A, z) -> E(G(A, z)) to ensure injective mapping and richer mode coverage.
• Use two encoders (E CNN or E ResNet) and two discriminators (two PatchGANs at different scales) with a U-Net generator, trained with Least Squares GAN loss and an L1 reconstruction term where applicable.
• Inject z into the generator via either add_to_input or add_to_all to explore different integration strategies.

Experimental results

Research questions

• RQ1 Can we model and sample from the conditional distribution p(B|A) to produce diverse outputs that remain faithful to A?
• RQ2 Does enforcing bidirectional consistency between latent codes and outputs reduce mode collapse while maintaining realism?
• RQ3 How do cVAE-GAN, cLR-GAN, and their combination (BicycleGAN) compare in terms of realism and diversity across standard multimodal translation tasks?
• RQ4 What is the impact of encoder architecture and latent code injection method on reconstruction and diversity?
• RQ5 How does latent code length affect diversity and realism across datasets?

Key findings

• BicycleGAN achieves higher realism and diversity than baseline methods, without suffering from mode collapse observed in some variants.
• Diversity measured by LPIPS is higher for cVAE-GAN, cVAE-GAN++, cLR-GAN, and BicycleGAN than baselines, while realism improves with the Hybrid objective.
• cLR-GAN alone can suffer from severe mode collapse (~15% identical outputs), whereas the full BicycleGAN avoids collapse and yields the best realism.
• Encoder choice affects latent encoding quality; E ResNet generally yields better latent reconstruction than E CNN, influencing performance.
• Two latent injection methods (add_to_input vs add_to_all) yield similar performance, with a preference for add_to_all in final results.
• Latent code length matters; very small z limits diversity, while very large z can hinder sampling; optimal length depends on dataset.