[Paper Review] A Variational Inequality Perspective on Generative Adversarial Networks

Gauthier Gidel, Hugo Berard+3 more|arXiv (Cornell University)|Feb 28, 2018

Generative Adversarial Networks and Image Synthesis56 references117 citations

TL;DR

The paper reframes GAN optimization as a variational inequality problem and introduces averaging, extrapolation, and extrapolation from the past to improve training stability and convergence, with theory and CIFAR-10 experiments showing gains.

ABSTRACT

Generative adversarial networks (GANs) form a generative modeling approach known for producing appealing samples, but they are notably difficult to train. One common way to tackle this issue has been to propose new formulations of the GAN objective. Yet, surprisingly few studies have looked at optimization methods designed for this adversarial training. In this work, we cast GAN optimization problems in the general variational inequality framework. Tapping into the mathematical programming literature, we counter some common misconceptions about the difficulties of saddle point optimization and propose to extend techniques designed for variational inequalities to the training of GANs. We apply averaging, extrapolation and a computationally cheaper variant that we call extrapolation from the past to the stochastic gradient method (SGD) and Adam.

Motivation & Objective

• Motivate GAN training as a two-player game and analyze it through variational inequality theory.
• Introduce averaging and extrapolation techniques from the VIP literature to GAN optimization.
• Provide convergence guarantees for extrapolation from the past in stochastic VIPs.
• Demonstrate practical improvements in GAN training on CIFAR-10 using WGAN-GP and ResNet generators.

Proposed method

• Cast GAN optimization (minimax/non-zero-sum formulations) as a variational inequality problem with operator F and domain Ω.
• Adapt VIP solution techniques—averaging and extrapolation (including extrapolation from the past)—to GAN training.
• Develop and analyze stochastic variants of SGD incorporating these VIP techniques (AvgSGD, AvgExtraSGD, AvgPastExtraSGD).
• Provide convergence results: linear convergence for extrapolation from the past under strong monotonicity and Lipschitz conditions; stochastic convergence rates with averaging.
• Show how these methods can be combined with Adam and SGD in practice (Extra-Adam, etc.).
• Offer code and implementation details for practical use.

Experimental results

Research questions

• RQ1 Can variational inequality techniques stabilize GAN training compared to standard SGD methods?
• RQ2 Do averaging and extrapolation reduce oscillations and improve convergence in GAN optimization?
• RQ3 What are the convergence guarantees for these VIP-based methods in both batch and stochastic settings?
• RQ4 How do VIP-based methods integrate with common optimizers like SGD and Adam in GAN contexts?
• RQ5 Do these methods yield measurable improvements on standard GAN benchmarks (e.g., CIFAR-10) with modern architectures?

Key findings

• Averaging and extrapolation reduce oscillations in GAN training by addressing the variational-inequality nature of the game.
• Extrapolation from the past achieves linear convergence for strongly monotone operators, with a favorable theoretical rate compared to standard SGD.
• In stochastic settings, AvgPastExtraSGD and related variants provide reduced variance and competitive convergence guarantees.
• Empirical results show 4–6% improvements on inception score and Fréchet inception distance on CIFAR-10 when using WGAN-GP with a ResNet generator.
• The work includes practical algorithm variants and code, enabling replication and integration with Adam/Sgd optimizers.