[Paper Review] Generalization and Equilibrium in Generative Adversarial Nets (GANs)

Sanjeev Arora, Rong Ge+3 more|arXiv (Cornell University)|Mar 2, 2017

Generative Adversarial Networks and Image Synthesis211 citations

TL;DR

The paper shows GAN generalization can fail under standard metrics but holds for neural-net distance, proves existence of approximate equilibria via generator mixtures, and introduces MIX+GAN as a practical stabilization technique.

ABSTRACT

We show that training of generative adversarial network (GAN) may not have good generalization properties; e.g., training may appear successful but the trained distribution may be far from target distribution in standard metrics. However, generalization does occur for a weaker metric called neural net distance. It is also shown that an approximate pure equilibrium exists in the discriminator/generator game for a special class of generators with natural training objectives when generator capacity and training set sizes are moderate. This existence of equilibrium inspires MIX+GAN protocol, which can be combined with any existing GAN training, and empirically shown to improve some of them.

Motivation & Objective

• Demonstrate that GAN training may not generalize under standard distribution distances.
• Introduce neural net distance as a tractable generalization metric for GANs.
• Prove existence of approximate pure equilibria via generator mixtures in GAN games.
• Propose and empirically validate MIX+GAN as a practical training framework.

Proposed method

• Define GAN training with neural network discriminators and generators and common objective functions.
• Introduce F-distance and neural net distance as unifying divergence measures.
• Prove generalization guarantees for neural net distance with polynomial sample complexity.
• Show that infinite mixtures of generators can achieve near-optimal equilibria in GAN games.
• Propose a finite-mixture construction to realize approximate pure equilibria and derive network-size bounds.
• Introduce MIX+GAN as a practical training protocol that combines mixtures with existing GAN objectives and employs entropy regularization to encourage diversity.

Experimental results

Research questions

• RQ1 Does GAN training generalize under standard distance metrics (JS, Wasserstein) with polynomial samples?
• RQ2 Can a weaker metric (neural net distance) provide generalization guarantees for GANs?
• RQ3 Do equilibria exist in GAN games, and can they be realized with practical network sizes?
• RQ4 Can mixtures of generators achieve approximate equilibria and can this be implemented as a single network (pure equilibrium)?
• RQ5 Does a MIX+GAN framework stabilize training and improve performance empirically?

Key findings

• Standard JS divergence and Wasserstein distance do not generalize with polynomial samples.
• Neural net distance enjoys generalization with polynomial samples, providing a proxy guarantee for GAN training.
• An approximate equilibrium exists via finite mixtures of generators, and a pure-equilibrium-like construction is possible with modest network-size increases.
• A MIX+GAN framework stabilizes training and yields improvements on datasets like CIFAR-10, MNIST, and CelebA.
• Empirical results show MIX+DCGAN achieving higher Inception Scores than several baselines.