[Paper Review] Unpaired Image-to-Image Translation using Adversarial Consistency Loss

Yihao Zhao, Ruihai Wu+1 more|arXiv (Cornell University)|Mar 10, 2020

Advanced Image Processing Techniques34 references4 citations

TL;DR

This paper proposes ACL-GAN, a novel unpaired image-to-image translation framework that replaces cycle-consistency loss with an adversarial-consistency loss to preserve essential features while enabling shape changes and object removal. By encouraging distribution-level similarity between source and translated images rather than pixel-level reconstruction, ACL-GAN achieves state-of-the-art results on glasses removal, male-to-female, and selfie-to-anime translation tasks with improved realism and diversity.

ABSTRACT

Unpaired image-to-image translation is a class of vision problems whose goal is to find the mapping between different image domains using unpaired training data. Cycle-consistency loss is a widely used constraint for such problems. However, due to the strict pixel-level constraint, it cannot perform geometric changes, remove large objects, or ignore irrelevant texture. In this paper, we propose a novel adversarial-consistency loss for image-to-image translation. This loss does not require the translated image to be translated back to be a specific source image but can encourage the translated images to retain important features of the source images and overcome the drawbacks of cycle-consistency loss noted above. Our method achieves state-of-the-art results on three challenging tasks: glasses removal, male-to-female translation, and selfie-to-anime translation.

Motivation & Objective

• To address the limitations of cycle-consistency loss in unpaired image-to-image translation, which forces pixel-level reconstruction and preserves unwanted features like glasses or beards.
• To enable realistic image translation that supports shape modifications, object removal, and multi-modal outputs without requiring paired training data.
• To develop a method that maintains domain-common features between source and target images through distribution-level alignment rather than pixel-level identity.
• To achieve higher-quality, more diverse, and more realistic translations compared to existing cycle-consistency-based and dual-generator methods.

Proposed method

• Introduces adversarial-consistency loss (ACL), which encourages the translated image to be similar to the source image in terms of shared domain features, without requiring exact pixel-level reconstruction.
• Uses a GAN-based framework with a generator and discriminator, where the generator translates images from source to target domain, and the discriminator distinguishes real from generated images.
• Employs a style-conditional generator architecture inspired by MUNIT, allowing disentangled content and style representations for multi-modal translation.
• Combines adversarial-consistency loss with standard adversarial loss and identity loss to stabilize training and improve feature retention.
• Relies on feature-level similarity via a pre-trained feature extractor (e.g., VGG) to compute consistency between source and translated images at intermediate layers.
• Does not require cycle-consistency or duplicate generators, reducing model complexity and computational cost compared to methods like CouncilGAN.

Experimental results

Research questions

• RQ1 Can adversarial-consistency loss effectively replace cycle-consistency loss in unpaired image translation while preserving essential features and enabling shape changes?
• RQ2 How does the proposed method perform on tasks requiring large structural changes, such as male-to-female face translation or selfie-to-anime style transfer?
• RQ3 Can ACL-GAN achieve better image quality and diversity than cycle-consistency-based models without using paired data?
• RQ4 Does the absence of cycle-consistency lead to artifacts or degradation in translation quality, and how does the method mitigate this?

Key findings

• ACL-GAN achieves state-of-the-art performance on three challenging unpaired image translation tasks: glasses removal, male-to-female face translation, and selfie-to-anime translation.
• On the male-to-female translation task, ACL-GAN generates more feminine faces with no beard and longer hair, outperforming cycle-consistency-based models like CycleGAN and MUNIT in both qualitative and quantitative evaluations.
• For selfie-to-anime translation, ACL-GAN produces more stylized and better-organized facial features (e.g., larger eyes) compared to baselines, even outperforming U-GAT-IT in light mode despite using fewer than half the parameters.
• The method successfully removes glasses and other large objects without leaving traces, unlike cycle-consistency-based models that preserve such features due to pixel-level reconstruction.
• ACL-GAN achieves higher-quality results than CouncilGAN, which uses duplicate generators and discriminators, due to explicit feature alignment via adversarial-consistency loss.
• Failure cases include incomplete removal of inconspicuous glasses and artifacts when input faces are small or obscured, indicating limitations in handling rare or complex input distributions.