[Paper Review] GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Martin Heusel, Hubert Ramsauer+3 more|arXiv (Cornell University)|Jun 26, 2017

Generative Adversarial Networks and Image SynthesisComputer Science4,490 citations

TL;DR

Introduces a two time-scale update rule (TTUR) for GANs with separate discriminator and generator learning rates, proves convergence to a local Nash equilibrium, analyzes Adam as heavy ball with friction, and proposes Fréchet Inception Distance (FID) for evaluation; TTUR improves GAN performance across multiple datasets.

ABSTRACT

Generative Adversarial Networks (GANs) excel at creating realistic images with complex models for which maximum likelihood is infeasible. However, the convergence of GAN training has still not been proved. We propose a two time-scale update rule (TTUR) for training GANs with stochastic gradient descent on arbitrary GAN loss functions. TTUR has an individual learning rate for both the discriminator and the generator. Using the theory of stochastic approximation, we prove that the TTUR converges under mild assumptions to a stationary local Nash equilibrium. The convergence carries over to the popular Adam optimization, for which we prove that it follows the dynamics of a heavy ball with friction and thus prefers flat minima in the objective landscape. For the evaluation of the performance of GANs at image generation, we introduce the "Fréchet Inception Distance" (FID) which captures the similarity of generated images to real ones better than the Inception Score. In experiments, TTUR improves learning for DCGANs and Improved Wasserstein GANs (WGAN-GP) outperforming conventional GAN training on CelebA, CIFAR-10, SVHN, LSUN Bedrooms, and the One Billion Word Benchmark.

Motivation & Objective

• Motivate and address the lack of convergence guarantees in GAN training.
• Propose a two time-scale update rule (TTUR) with separate learning rates for discriminator and generator.
• Provide theoretical convergence results to stationary local Nash equilibria under TTUR and analyze Adam dynamics as heavy ball with friction.
• Introduce Fréchet Inception Distance (FID) as a robust GAN evaluation metric.
• Empirically validate TTUR on multiple GAN variants and datasets.

Proposed method

• Define TTUR with learning rates b(n) for the discriminator and a(n) for the generator and derive stochastic approximation updates.
• Prove convergence of TTUR to a stationary local Nash equilibrium under mild assumptions using ODE and stochastic approximation theory.
• Describe Adam as a heavy ball with friction (HBF) and connect it to TTUR convergence.
• Show that Adam with TTUR preserves convergence by linking its dynamics to HBF ODEs.
• Introduce Fréchet Inception Distance (FID) as a means to evaluate how close generated data are to real data using mean and covariance in Inception coding space.
• Experiment with DCGAN, WGAN-GP (image and language data), and One Billion Word Benchmark to compare TTUR against standard one-time-scale training.

Experimental results

Research questions

• RQ1 Can GANs trained with TTUR converge to a stationary local Nash equilibrium under stochastic gradient updates?
• RQ2 How do separate learning rates for the discriminator and generator affect convergence and performance compared to single-time-scale training?
• RQ3 Is Adam compatible with TTUR in terms of convergence, and how does its dynamics influence minima quality?
• RQ4 Does FID provide a more reliable evaluation of GAN quality than Inception Score across varied perturbations and datasets?
• RQ5 Do TTUR-trained GANs outperform conventional training on image and language benchmarks?

Key findings

• TTUR converges to a stationary local Nash equilibrium under mild assumptions.
• Adam can be interpreted as heavy ball with friction, and with TTUR it follows dynamics that preserve convergence.
• TTUR combined with Adam yields convergence and tends to prefer flat minima.
• TTUR consistently improves performance over standard single-time-scale GAN training across CelebA, CIFAR-10, SVHN, LSUN Bedrooms, and One Billion Word benchmarks.
• FID correlates better with data disturbances and human judgment than Inception Score, demonstrating robust GAN evaluation.
• TTUR reduces training oscillations and variance, leading to more stable learning than original methods.