[Paper Review] Which Training Methods for GANs do actually Converge?

Lars Mescheder, Andreas Geiger+1 more|arXiv (Cornell University)|Jan 13, 2018

Generative Adversarial Networks and Image Synthesis606 citations

TL;DR

The paper analyzes GAN training dynamics, showing unregularized GANs may not converge when data/generator lie on manifolds, and demonstrates which regularizers (instance noise, zero-centered gradient penalties) promote local convergence, while WGANs/WGAN-GP may fail under certain update schemes.

ABSTRACT

Recent work has shown local convergence of GAN training for absolutely continuous data and generator distributions. In this paper, we show that the requirement of absolute continuity is necessary: we describe a simple yet prototypical counterexample showing that in the more realistic case of distributions that are not absolutely continuous, unregularized GAN training is not always convergent. Furthermore, we discuss regularization strategies that were recently proposed to stabilize GAN training. Our analysis shows that GAN training with instance noise or zero-centered gradient penalties converges. On the other hand, we show that Wasserstein-GANs and WGAN-GP with a finite number of discriminator updates per generator update do not always converge to the equilibrium point. We discuss these results, leading us to a new explanation for the stability problems of GAN training. Based on our analysis, we extend our convergence results to more general GANs and prove local convergence for simplified gradient penalties even if the generator and data distribution lie on lower dimensional manifolds. We find these penalties to work well in practice and use them to learn high-resolution generative image models for a variety of datasets with little hyperparameter tuning.

Motivation & Objective

• Assess whether GAN training converges under general (non-absolutely continuous) data and generator distributions.
• Evaluate how regularization strategies affect convergence in simple and realistic GAN settings.
• Propose simplified gradient penalties with provable local convergence guarantees.
• Understand practical implications of instability sources in GAN training for high-resolution image generation.

Proposed method

• Introduce the Dirac-GAN as a minimal counterexample to analyze convergence properties of GAN training.
• Analyze the eigenvalues of the Jacobian of the gradient field at equilibrium for various training schemes.
• Examine several regularization techniques (instance noise, zero-centered gradient penalties, consensus optimization) and derive their impact on local convergence.
• Derive simplified gradient penalty forms and prove local convergence under specified assumptions.
• Extend convergence analysis to more general GANs with distributions on lower-dimensional manifolds.
• Conduct 2D and image-domain experiments to compare convergence behavior across methods.

Experimental results

Research questions

• RQ1 Do unregularized GANs converge when data and generator distributions lie on lower-dimensional manifolds?
• RQ2 Which regularization techniques restore local convergence in GAN training?
• RQ3 Under what conditions do Wasserstein-based GANs and gradient-penalized variants converge or fail to converge?
• RQ4 Can simplified gradient penalties provide provable convergence in more general GANs?
• RQ5 How do convergence properties translate from the Dirac-GAN to high-dimensional image generation tasks?

Key findings

• Unregularized GAN training can be non-convergent in the general case with manifold-supported distributions.
• WGANs and WGAN-GP with a fixed number of discriminator updates per generator update do not universally converge on the Dirac-GAN.
• Instance noise and zero-centered gradient penalties induce local convergence in the Dirac-GAN.
• Gradient penalties with simplified forms can achieve local convergence near equilibrium under suitable assumptions.
• Regularized penalties extend convergence results to GANs where generator and data distributions lie on lower-dimensional manifolds.
• Empirical results show R1/R2 gradient penalties stabilize training and enable high-resolution image generation with limited hyperparameter tuning.