[논문 리뷰] A Variational Inequality Perspective on Generative Adversarial Networks

Gauthier Gidel, Hugo Berard외 3명|arXiv (Cornell University)|2018. 02. 28.

Generative Adversarial Networks and Image Synthesis참고 문헌 56인용 수 117

한 줄 요약

본 논문은 GAN 최적화를 변분부정(variational inequality) 문제로 재구성하고, 안정적인 학습 및 수렴을 개선하기 위해 평균화(averaging), 외삽(extrapolation), 과거로부터의 extrapolation을 도입하며, 이론과 CIFAR-10 실험에서 이점을 보인다.

ABSTRACT

Generative adversarial networks (GANs) form a generative modeling approach known for producing appealing samples, but they are notably difficult to train. One common way to tackle this issue has been to propose new formulations of the GAN objective. Yet, surprisingly few studies have looked at optimization methods designed for this adversarial training. In this work, we cast GAN optimization problems in the general variational inequality framework. Tapping into the mathematical programming literature, we counter some common misconceptions about the difficulties of saddle point optimization and propose to extend techniques designed for variational inequalities to the training of GANs. We apply averaging, extrapolation and a computationally cheaper variant that we call extrapolation from the past to the stochastic gradient method (SGD) and Adam.

연구 동기 및 목표

• Motivate GAN training as a two-player game and analyze it through variational inequality theory.
• Introduce averaging and extrapolation techniques from the VIP literature to GAN optimization.
• Provide convergence guarantees for extrapolation from the past in stochastic VIPs.
• Demonstrate practical improvements in GAN training on CIFAR-10 using WGAN-GP and ResNet generators.

제안 방법

• Cast GAN optimization (minimax/non-zero-sum formulations) as a variational inequality problem with operator F and domain Ω.
• Adapt VIP solution techniques—averaging and extrapolation (including extrapolation from the past)—to GAN training.
• Develop and analyze stochastic variants of SGD incorporating these VIP techniques (AvgSGD, AvgExtraSGD, AvgPastExtraSGD).
• Provide convergence results: linear convergence for extrapolation from the past under strong monotonicity and Lipschitz conditions; stochastic convergence rates with averaging.
• Show how these methods can be combined with Adam and SGD in practice (Extra-Adam, etc.).
• Offer code and implementation details for practical use.

실험 결과

연구 질문

• RQ1 Can variational inequality techniques stabilize GAN training compared to standard SGD methods?
• RQ2 Do averaging and extrapolation reduce oscillations and improve convergence in GAN optimization?
• RQ3 What are the convergence guarantees for these VIP-based methods in both batch and stochastic settings?
• RQ4 How do VIP-based methods integrate with common optimizers like SGD and Adam in GAN contexts?
• RQ5 Do these methods yield measurable improvements on standard GAN benchmarks (e.g., CIFAR-10) with modern architectures?

주요 결과

• Averaging and extrapolation reduce oscillations in GAN training by addressing the variational-inequality nature of the game.
• Extrapolation from the past achieves linear convergence for strongly monotone operators, with a favorable theoretical rate compared to standard SGD.
• In stochastic settings, AvgPastExtraSGD and related variants provide reduced variance and competitive convergence guarantees.
• Empirical results show 4–6% improvements on inception score and Fréchet inception distance on CIFAR-10 when using WGAN-GP with a ResNet generator.
• The work includes practical algorithm variants and code, enabling replication and integration with Adam/Sgd optimizers.