[Paper Review] A Closer Look at the Optimization Landscapes of Generative Adversarial Networks

Hugo Berard, Gauthier Gidel+3 more|arXiv (Cornell University)|Jun 11, 2019

Generative Adversarial Networks and Image Synthesis41 references17 citations

TL;DR

This paper introduces a novel visualization technique called Path-angle to analyze the game vector field in Generative Adversarial Networks (GANs), revealing that training dynamics exhibit significant rotational behavior around locally stable stationary points (LSSPs). Despite not converging to local Nash equilibria, GANs achieve high performance by settling at saddle points in the generator loss landscape, suggesting that Nash equilibrium is not necessary for good generative results.

ABSTRACT

Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks. In this paper, we propose new visualization techniques for the optimization landscapes of GANs that enable us to study the game vector field resulting from the concatenation of the gradient of both players. Using these visualization techniques we try to bridge the gap between theory and practice by showing empirically that the training of GANs exhibits significant rotations around Local Stable Stationary Points (LSSP), similar to the one predicted by theory on toy examples. Moreover, we provide empirical evidence that GAN training converge to a stable stationary point which is a saddle point for the generator loss, not a minimum, while still achieving excellent performance.

Motivation & Objective

• To bridge the gap between theoretical predictions of rotational dynamics in GANs and empirical observations in practical training settings.
• To investigate whether rotational behavior around locally stable stationary points (LSSPs) occurs in real-world GAN training on datasets like MNIST and CIFAR10.
• To determine whether existing GAN training methods converge to local Nash equilibria or to saddle points in the generator loss landscape.
• To evaluate how different GAN objectives (e.g., NSGAN vs. WGAN-GP) shape the optimization landscape and influence training stability.

Proposed method

• Proposes a new visualization tool called Path-angle to analyze the game vector field by capturing rotational and attractive behaviors near stationary points in high-dimensional GAN optimization.
• Empirically studies the game vector field using gradient concatenation of both generator and discriminator, focusing on local stability and rotational dynamics.
• Analyzes the Hessian eigenvalues of both players at convergence points to assess whether the solution is a local minimum, saddle point, or local Nash equilibrium.
• Applies the method across multiple GAN architectures (NSGAN, WGAN-GP), optimizers (Adam, ExtraAdam), and datasets (Mixture of Gaussians, MNIST, CIFAR10).
• Uses top-k Hessian eigenvalues (by magnitude) to infer local geometry without full spectral computation, enabling scalability to deep networks.
• Compares eigenvalue distributions of the Hessian for generator and discriminator across different GAN objectives to assess sharpness and regularization effects.

Experimental results

Research questions

• RQ1 Does rotational behavior in the game vector field—predicted by theory in toy examples—actually occur during training of GANs on real-world datasets?
• RQ2 Do standard GAN training procedures converge to local Nash equilibria, or do they instead settle at saddle points in the generator loss landscape?
• RQ3 How do different GAN objectives (e.g., NSGAN vs. WGAN-GP) affect the geometry of the optimization landscape, particularly in terms of rotational dynamics and solution sharpness?
• RQ4 Is convergence to a local Nash equilibrium necessary for high-quality generative performance in GANs?

Key findings

• GAN training exhibits significant rotational dynamics around locally stable stationary points (LSSPs), confirming theoretical predictions in practical, high-dimensional settings.
• Despite strong performance on evaluation metrics (e.g., IS = 9.46 on MNIST for WGAN-GP), the generator converges to a saddle point in its loss landscape, not a local minimum.
• The discriminator in NSGAN converges to a solution with much larger positive Hessian eigenvalues than in WGAN-GP, indicating a sharper minimum, which is mitigated by the gradient penalty in WGAN-GP.
• WGAN-GP shows a more balanced distribution of complex eigenvalues in the Hessian, with a tendency to spread toward the right half of the imaginary axis, suggesting more stable dynamics compared to NSGAN.
• The generator loss landscape at convergence is dominated by negative eigenvalues, confirming that the solution is a saddle point, not a local minimum.
• These findings suggest that convergence to a local Nash equilibrium is not required for high-quality generation, and that rotational dynamics are a key challenge in GAN optimization.