[Paper Review] Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Jun-Yan Zhu, Taesung Park+2 more|arXiv (Cornell University)|Mar 30, 2017

Generative Adversarial Networks and Image Synthesis65 references2,309 citations

TL;DR

CycleGAN learns mappings between unpaired image domains using adversarial losses with cycle consistency, enabling translation between domains like Monet paintings and photographs without paired data.

ABSTRACT

Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain $X$ to a target domain $Y$ in the absence of paired examples. Our goal is to learn a mapping $G: X ightarrow Y$ such that the distribution of images from $G(X)$ is indistinguishable from the distribution $Y$ using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping $F: Y ightarrow X$ and introduce a cycle consistency loss to push $F(G(X)) \approx X$ (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.

Motivation & Objective

• Address the lack of paired training data for image-to-image translation by learning mappings between two domains X and Y.
• Impose cycle-consistency to regularize mappings G: X->Y and F: Y->X so that F(G(X))≈X and G(F(Y))≈Y.
• Leverage adversarial losses to align translated images with target domain distributions while maintaining structure via cycle loss.
• Demonstrate versatility across tasks such as collection style transfer, object transfiguration, season transfer, and photo enhancement.

Proposed method

• Use two generators G:X->Y and F:Y->X with corresponding discriminators D_Y and D_X.
• Apply adversarial losses to make G(X) indistinguishable from Y and F(Y) indistinguishable from X.
• Introduce cycle-consistency losses L_cyc that minimize ||F(G(x)) - x||_1 and ||G(F(y)) - y||_1.
• Combine into full objective L = L_GAN(G,D_Y,X,Y) + L_GAN(F,D_X,Y,X) + λ L_cyc(G,F) with λ=10.
• Adopt architectural choices from Johnson et al. for generators, PatchGAN discriminators, and instance normalization.
• Stabilize training with least-squares GAN loss and a history of generated images to train discriminators.

Experimental results

Research questions

• RQ1 Can unpaired image collections be translated across domains without paired examples while preserving meaningful content?
• RQ2 Does cycle-consistency sufficiently regularize the under-constrained translation problem to produce high-quality, realistic mappings?
• RQ3 How do CycleGAN translations compare to supervised baselines and other unpaired methods across diverse tasks?
• RQ4 What is the impact of the adversarial and cycle-consistency terms on translation quality?

Key findings

• CycleGAN achieves compelling translations on unpaired data, often matching supervised pix2pix quality without paired training.
• On maps and aerial photos, CycleGAN fooled human evaluators about real vs fake in ~26.8% and ~23.2% of trials at 256×256 resolution.
• For Cityscapes tasks, CycleGAN outperformed several baselines on labels-to-photo and photo-to-label metrics, approaching or exceeding some weakly supervised baselines.
• Ablation studies show removing either the GAN loss or the cycle-consistency loss degrades performance, with the full CycleGAN providing the best results.
• The method generalizes to multiple applications, including collection style transfer, object transfiguration, season transfer, and photo enhancement.