[Paper Review] MIXGAN: Learning Concepts from Different Domains for Mixture Generation

Guang-Yuan Hao, Hong-Xing Yu+1 more|arXiv (Cornell University)|Jul 4, 2018

Generative Adversarial Networks and Image Synthesis27 references3 citations

TL;DR

MIXGAN proposes a novel generative adversarial network that learns content concepts from one domain and style concepts from another, enabling the synthesis of images in a new domain by fusing these distinct concepts. By using a hierarchical content decoder and a style-embedded mixture decoder, MIXGAN achieves superior performance in mixture generation, with human evaluation showing a 70% success rate—significantly outperforming AAE, LSGAN, and CycleGAN, which scored below 10%.

ABSTRACT

In this work, we present an interesting attempt on mixture generation: absorbing different image concepts (e.g., content and style) from different domains and thus generating a new domain with learned concepts. In particular, we propose a mixture generative adversarial network (MIXGAN). MIXGAN learns concepts of content and style from two domains respectively, and thus can join them for mixture generation in a new domain, i.e., generating images with content from one domain and style from another. MIXGAN overcomes the limitation of current GAN-based models which either generate new images in the same domain as they observed in training stage, or require off-the-shelf content templates for transferring or translation. Extensive experimental results demonstrate the effectiveness of MIXGAN as compared to related state-of-the-art GAN-based models.

Motivation & Objective

• To address the limitation of existing GANs that generate only within the same domain as training data.
• To overcome the need for pre-existing content templates in style transfer and image-to-image translation models.
• To enable the generation of new images that combine content from one domain and style from another, forming a novel, hybrid domain.
• To develop a framework that explicitly learns and fuses distinct concepts (content and style) from different domains for creative image synthesis.

Proposed method

• MIXGAN employs a mixture generator with a content decoder that extracts hierarchical features from the content domain (e.g., shape of bags).
• A mixture decoder learns the style concept from the style domain (e.g., color patterns of shoes) and is designed to progressively integrate the style into the content features.
• The generator uses noise as a latent code to initiate generation, analogous to purchasing ingredients before cooking.
• The model uses a conditional adversarial training scheme where the discriminator distinguishes real images from generated ones, encouraging realistic output.
• A deep binary classifier is used to assign each generated sample to the nearest training distribution for MMD-based evaluation.
• The framework is trained end-to-end using adversarial loss and reconstruction loss to preserve content and style fidelity.

Experimental results

Research questions

• RQ1 Can a GAN-based model generate images that combine content from one domain and style from another without relying on pre-existing content templates?
• RQ2 How can content and style concepts be disentangled and jointly learned from different domains for effective mixture generation?
• RQ3 Can the model generate samples that belong to a new domain, distinct from both training distributions, as evidenced by distributional analysis?
• RQ4 To what extent does the model outperform existing GANs in generating conceptually blended images, as measured by human evaluation and MMD distance?

Key findings

• MIXGAN achieved a human evaluation success rate of over 70% in generating images that combine content and style from two different domains, significantly outperforming AAE, LSGAN, and CycleGAN, which scored below 10%.
• t-SNE visualization confirmed that MIXGAN-generated samples lie in a distribution that is 'between' the two training distributions, indicating successful mixture generation.
• The MMD distance between generated samples and their nearest training set was higher for MIXGAN than for baselines, suggesting that generated images are not simply copies of training data.
• The model successfully generated colorful, type-script digits by combining the content of handwritten digits and the style of type-script digits, demonstrating cross-domain fusion.
• In tasks involving bags and shoes, MIXGAN produced visually coherent images where both content (bag shape) and style (shoe color patterns) were perceptually blended.
• The ablation study confirmed that the mixture generator’s design—separating content and style learning and integrating them via a hierarchical decoder—was essential for performance.