[Paper Review] Conditional Generative Adversarial Nets

Mehdi Mirza, Simon Osindero|arXiv (Cornell University)|Nov 6, 2014

Generative Adversarial Networks and Image Synthesis15 references8,878 citations

TL;DR

Introduces conditional GANs by feeding conditioning information to both generator and discriminator, demonstrating MNIST digit generation conditioned on labels and multimodal tagging on MIR Flickr data.

ABSTRACT

Generative Adversarial Nets [8] were recently introduced as a novel way to train generative models. In this work we introduce the conditional version of generative adversarial nets, which can be constructed by simply feeding the data, y, we wish to condition on to both the generator and discriminator. We show that this model can generate MNIST digits conditioned on class labels. We also illustrate how this model could be used to learn a multi-modal model, and provide preliminary examples of an application to image tagging in which we demonstrate how this approach can generate descriptive tags which are not part of training labels.

Motivation & Objective

• Motivate conditioning in GANs to control data generation and explore multi-modal modeling.
• Show how to integrate auxiliary information y into G and D to steer generation.
• Demonstrate conditional GANs on MNIST digits conditioned on class labels.
• Demonstrate conditional GANs for multi-modal learning using image features and text tags.

Proposed method

• Extend GANs by conditioning both generator and discriminator on auxiliary information y.
• Combine z (noise) and y in a joint hidden representation within the generator.
• Feed x and y into the discriminator to estimate the probability of data vs generated samples.
• Train with a two-player minimax objective V(D,G) = E[x~pdata] [log D(x|y)] + E[z~pz] [log(1 - D(G(z|y)))]
• Use ReLU and maxout units in network architectures to map inputs to a shared hidden representation before final output.
• Evaluate with Parzen window log-likelihood estimates on MNIST and perform multimodal tagging on MIR Flickr with conditional generation of tag vectors.

Experimental results

Research questions

• RQ1 Can conditioning GANs on auxiliary information y direct the generation process to produce data corresponding to that conditioning?
• RQ2 Can a conditional GAN learn multi-modal or descriptive tag distributions from image features and text representations?
• RQ3 How does conditioning affect sample quality and log-likelihood estimates on MNIST compared to unconditioned GANs?

Key findings

• Conditional GANs can generate MNIST digits conditioned on class labels (one-hot encoded).
• Parzen window log-likelihood on MNIST shows conditional nets achieving 132±1.8 for MNIST vs 225±2 for standard adversarial nets in the reported setup.
• The model demonstrates a preliminary multimodal learning capability by generating tag vectors conditioned on image features and language representations.
• Generated tags for MIR Flickr examples show plausible, varied descriptions aligned with image content.
• The authors present results as proof-of-concept with potential for further hyperparameter tuning to match or exceed non-conditional GANs.