[Paper Review] Contrastive Learning for Unpaired Image-to-Image Translation

Taesung Park, Alexei A. Efros+2 more|arXiv (Cornell University)|Jul 30, 2020

Generative Adversarial Networks and Image SynthesisComputer Science89 references126 citations

TL;DR

The paper introduces a one-sided, unpaired image-to-image translation method (CUT) that maximizes mutual information between input and output patches using a multilayer, patchwise contrastive loss (PatchNCE), enabling high-quality translations without cycle-consistency or inverse mappings.

ABSTRACT

In image-to-image translation, each patch in the output should reflect the content of the corresponding patch in the input, independent of domain. We propose a straightforward method for doing so -- maximizing mutual information between the two, using a framework based on contrastive learning. The method encourages two elements (corresponding patches) to map to a similar point in a learned feature space, relative to other elements (other patches) in the dataset, referred to as negatives. We explore several critical design choices for making contrastive learning effective in the image synthesis setting. Notably, we use a multilayer, patch-based approach, rather than operate on entire images. Furthermore, we draw negatives from within the input image itself, rather than from the rest of the dataset. We demonstrate that our framework enables one-sided translation in the unpaired image-to-image translation setting, while improving quality and reducing training time. In addition, our method can even be extended to the training setting where each "domain" is only a single image.

Motivation & Objective

• Motivate content preservation across domains without relying on cycle-consistency or bijections.
• Propose a contrastive learning objective that maximizes mutual information between corresponding input-output patches.
• Show that internal (within-image) negatives yield stronger signals for content preservation.
• Develop a lightweight, one-sided translation pipeline (CUT) that can even operate on single-image pairs.

Proposed method

• Use an encoder G_enc and decoder G_dec to form a two-part generator.
• Apply a multilayer, patchwise InfoNCE loss (PatchNCE) to align corresponding input-output patches across multiple encoder layers.
• Sample negatives from the same input image (internal negatives) to form an (N+1)-way classification with a temperature tau for the InfoNCE loss.
• Optionally include an external negative sampling variant (external NCE) using a moving-average encoder, though internal negatives perform best.
• Replace cycle-consistency with a GAN loss plus PatchNCE loss on input (and optionally on output) to enforce domain realism and content preservation.
• Provide a faster variant (FastCUT) by omitting the identity-preserving term and simplifying training while maintaining performance.

Experimental results

Research questions

• RQ1 Can maximizing mutual information between corresponding input and output patches enable content-preserving translation without cycle-consistency?
• RQ2 Are internal (within-image) negatives more effective than external negatives for learning cross-domain patch correspondences?
• RQ3 How do multi-layer, patchwise contrastive objectives compare to traditional cycle-consistency-based methods in unpaired translation?
• RQ4 Can unpaired translation be trained effectively from a single image per domain (one-shot/single-image translation) using PatchNCE?
• RQ5 What are the trade-offs in training time, memory, and quality between CUT and its faster variant FastCUT?

Key findings

• The proposed PatchNCE-based loss yields higher quality translations than several baselines (CycleGAN, MUNIT, DRIT, DistanceGAN, GcGAN) across datasets like Horse→Zebra and Cityscapes.
• Internal negatives from the input image outperform external negatives drawn from other images or a memory bank.
• Using multiple encoder layers for patch-level contrastive learning improves content preservation and translation quality versus using only the last layer.
• The identity-regularized variant with PatchNCE on the output domain (Y) provides training stability, reducing collapse risk.
• CUT achieves faster training and lower memory usage compared to CycleGAN, with competitive or superior FID and semantic correspondence metrics; FastCUT further reduces computational cost with comparable results.
• The method can be extended to single-image translation tasks (SinCUT), enabling high-resolution painting-to-photo style transfers using only one example per domain.