[Paper Review] Unpaired Multi-Domain Image Generation via Regularized Conditional GANs

Xudong Mao, Qing Li|arXiv (Cornell University)|May 7, 2018

Generative Adversarial Networks and Image Synthesis25 references4 citations

TL;DR

This paper proposes Regularized Conditional GAN (RegCGAN), a framework for unpaired multi-domain image generation that learns correspondence between domains without paired training data. By introducing two regularizers—one in the generator's first layer to enforce shared high-level semantics and another in the discriminator's final layer to produce invariant feature representations—it achieves state-of-the-art performance in generating aligned image pairs across diverse domains such as edges-to-photos, faces with attributes, and domain adaptation tasks.

ABSTRACT

In this paper, we study the problem of multi-domain image generation, the goal of which is to generate pairs of corresponding images from different domains. With the recent development in generative models, image generation has achieved great progress and has been applied to various computer vision tasks. However, multi-domain image generation may not achieve the desired performance due to the difficulty of learning the correspondence of different domain images, especially when the information of paired samples is not given. To tackle this problem, we propose Regularized Conditional GAN (RegCGAN) which is capable of learning to generate corresponding images in the absence of paired training data. RegCGAN is based on the conditional GAN, and we introduce two regularizers to guide the model to learn the corresponding semantics of different domains. We evaluate the proposed model on several tasks for which paired training data is not given, including the generation of edges and photos, the generation of faces with different attributes, etc. The experimental results show that our model can successfully generate corresponding images for all these tasks, while outperforms the baseline methods. We also introduce an approach of applying RegCGAN to unsupervised domain adaptation.

Motivation & Objective

• Address the challenge of multi-domain image generation when paired training data is unavailable.
• Overcome the failure of standard conditional GANs in learning cross-domain correspondence without paired examples.
• Develop a method that learns shared high-level semantics across domains while preserving domain-specific characteristics.
• Enable application of the model to unsupervised domain adaptation using invariant feature representations.
• Improve generalization and performance in zero-shot image translation and domain adaptation tasks.

Proposed method

• Condition the generator and discriminator on domain-specific variables, while sharing a common latent code to encode shared semantics.
• Introduce a first-layer generator regularizer that penalizes L2 distance between outputs of identical latent codes but different domain codes, enforcing semantic consistency.
• Add a final-layer discriminator regularizer that encourages similar loss outputs for corresponding image pairs, guiding the generator to produce aligned images.
• Use the discriminator’s last hidden layer as a feature extractor with invariant representations across domains, enabling transfer learning.
• Jointly train the generator and discriminator with adversarial loss and the two regularizers using a weighted sum of losses.
• Apply the learned invariant features to unsupervised domain adaptation by attaching a classifier to the discriminator’s final layer.

Experimental results

Research questions

• RQ1 Can a conditional GAN be regularized to learn cross-domain correspondence without paired training data?
• RQ2 How can shared high-level semantics be enforced across different domains in the absence of paired examples?
• RQ3 Can the discriminator’s final layer produce invariant representations that generalize across domains for downstream tasks?
• RQ4 Does the proposed regularized framework outperform existing methods in unpaired image translation and domain adaptation?
• RQ5 To what extent can the learned features support unsupervised domain adaptation with minimal labeled data?

Key findings

• RegCGAN successfully generates corresponding image pairs across diverse domains—including edges-to-photos, face attributes, Monet-style images, and seasonal changes—without any paired training data.
• On the MNIST-to-USPS and USPS-to-MNIST domain adaptation tasks, RegCGAN achieved 93.1% and 89.5% accuracy on sampled target sets, respectively, outperforming DANN, ADDA, and CoGAN.
• On standard test sets, RegCGAN achieved 90.1% and 88.8% accuracy for MNIST-to-USPS and USPS-to-MNIST, respectively, demonstrating superior generalization over baselines.
• The model produces smooth interpolations in latent space, showing consistent semantic transitions in both content and style across generated image pairs.
• The invariant feature representations learned by the discriminator’s last layer enabled effective transfer learning, validating the method’s utility in unsupervised domain adaptation.
• Ablation studies confirmed that both regularizers are essential, as removing either significantly degraded performance on image translation and domain adaptation tasks.