[Paper Review] How to Train Your DRAGAN

Naveen Kodali, Jacob Abernethy+2 more|arXiv (Cornell University)|May 19, 2017

Generative Adversarial Networks and Image Synthesis7 references34 citations

TL;DR

This paper introduces DRAGAN, a GAN training algorithm that leverages regret minimization to stabilize training and avoid mode collapse by addressing spurious local equilibria in non-convex games. It achieves state-of-the-art performance across MNIST, CIFAR-10, and CelebA with minimal hyperparameter tuning and outperforms recent Wasserstein GAN variants.

ABSTRACT

Generative Adversarial Networks have emerged as an effective technique for estimating data distributions. The basic setup consists of two deep networks playing against each other in a zero-sum game setting. However, it is not understood if the networks reach an equilibrium eventually and what dynamics makes this possible. The current GAN training procedure, which involves simultaneous gradient descent, lacks a clear game-theoretic justification in the literature. In this paper, we introduce regret minimization as a technique to reach equilibrium in games and use this to justify the success of simultaneous GD in GANs. In addition, we present a hypothesis that mode collapse, which is a common occurrence in GAN training, happens due to the existence of spurious local equilibria in non-convex games. Motivated by these insights, we develop an algorithm called DRAGAN that is fast, simple to implement and achieves competitive performance in a stable fashion across different architectures (150 random setups), datasets (MNIST, CIFAR-10, and CelebA), and divergence measures with almost no hyperparameter tuning. We show significant improvements over the recently proposed Wasserstein GAN variants.

Motivation & Objective

• To understand the theoretical underpinnings of GAN training, particularly why simultaneous gradient descent succeeds despite lacking a game-theoretic justification.
• To investigate the causes of mode collapse in GANs, hypothesizing it arises from spurious local equilibria in non-convex games.
• To develop a simple, robust training algorithm that achieves stable, high-quality generation across diverse architectures, datasets, and divergence measures.
• To improve upon existing WGAN variants by reducing hyperparameter sensitivity and enhancing training stability and performance.

Proposed method

• Applies regret minimization techniques from game theory to justify and improve the simultaneous gradient descent process in GANs.
• Proposes DRAGAN as a training algorithm that modifies the GAN objective to reduce the likelihood of converging to spurious local equilibria.
• Employs a simple architectural modification that stabilizes training dynamics without requiring complex hyperparameter tuning.
• Uses a combination of gradient updates and regret-based optimization to encourage exploration and avoid mode collapse.
• Validates the approach across 150 random training setups on MNIST, CIFAR-10, and CelebA using multiple divergence measures.

Experimental results

Research questions

• RQ1 Why does simultaneous gradient descent in GANs lead to stable training despite lacking a formal game-theoretic foundation?
• RQ2 What causes mode collapse in GANs, and can it be attributed to spurious local equilibria in non-convex games?
• RQ3 Can regret minimization be effectively applied to GAN training to improve stability and performance?
• RQ4 How does DRAGAN compare to state-of-the-art WGAN variants in terms of sample quality and training stability?

Key findings

• DRAGAN achieves competitive performance across MNIST, CIFAR-10, and CelebA with almost no hyperparameter tuning.
• The algorithm demonstrates stable training across 150 random architectural and hyperparameter setups, indicating robustness.
• DRAGAN significantly outperforms recently proposed Wasserstein GAN variants in terms of training stability and sample quality.
• The use of regret minimization helps avoid spurious local equilibria, reducing the incidence of mode collapse in GAN training.