[Paper Review] Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data

Amjad Almahairi, Sai Rajeswar+3 more|arXiv (Cornell University)|Feb 27, 2018

Domain Adaptation and Few-Shot Learning28 references315 citations

TL;DR

Augmented CycleGAN learns many-to-many cross-domain mappings from unpaired data by augmenting domains with latent codes and enforcing cycle-consistency in augmented space, enabling diverse outputs per input. It outperforms cycles with deterministic mappings on several tasks and supports semi-supervised learning with attribute-face translation.

ABSTRACT

Learning inter-domain mappings from unpaired data can improve performance in structured prediction tasks, such as image segmentation, by reducing the need for paired data. CycleGAN was recently proposed for this problem, but critically assumes the underlying inter-domain mapping is approximately deterministic and one-to-one. This assumption renders the model ineffective for tasks requiring flexible, many-to-many mappings. We propose a new model, called Augmented CycleGAN, which learns many-to-many mappings between domains. We examine Augmented CycleGAN qualitatively and quantitatively on several image datasets.

Motivation & Objective

• Address the limitation of CycleGAN’s deterministic one-to-one mappings in cross-domain translation.
• Propose Augmented CycleGAN to model many-to-many mappings via auxiliary latent spaces.
• Demonstrate diverse output generation for a given input across varied domains.
• Show applicability to substantially different domains and semi-supervised learning with paired data.

Proposed method

• Introduce G_{A→B}(a,z_b) and G_{B→A}(b,z_a) acting on augmented spaces A×Z_b and B×Z_a.
• Use encoders E_A and E_B to map (A,B) pairs to latent codes z_a and z_b.
• Apply adversarial losses to match generated pairs and priors for (B, Z_a) and (A, Z_b) separately (marginal matching).
• Enforce cycle-consistency on augmented space via reconstructions a′ and z_b′, ensuring information preservation through latent codes.
• Incorporate two-cycle losses: reconstruct a from (a,z_b) and z_b from (a,z_b) to maximize mutual information I(b,z_b|a).
• Optionally extend to semi-supervised training with available paired data (a,b) to supervise G_{A→B} and G_{B→A}.

Experimental results

Research questions

• RQ1 Can many-to-many cross-domain mappings be learned from unpaired data by augmenting domains with latent codes?
• RQ2 Do augmented cycle-consistency losses preserve information and enable diverse outputs for a given input?
• RQ3 How does Augmented CycleGAN perform on disparate domain pairs (edges→photos, faces→attributes) compared to CycleGAN and stochastic variants?
• RQ4 Can the model be effectively trained in semi-supervised settings with limited paired data?
• RQ5 What architectural choices (e.g., Conditional Normalization) improve diversity and stability of stochastic mappings?

Key findings

• AugCGAN produces diverse outputs for a given input by sampling latent codes z_b and z_a.
• On edges-to-photos, AugCGAN achieves lower reconstruction error (L1) than CycleGAN and Stochastic CycleGAN, especially with 10% paired data.
• AugCGAN attains lower MSE for predicting edges from shoes than Δ-GAN with comparable architecture, and improves further with supervision.
• Qualitative results show AugCGAN maintaining diversity across generated images and avoiding the mode-collapse observed in Stochastic CycleGAN.
• In CelebA attribute-to-face tasks, AugCGAN achieves competitive or superior precision and nDCG in semi-supervised settings compared to baselines.
• The model preserves realistic appearance while enabling meaningful variations through latent conditioning, outperforming baselines in diversity metrics.