[論文レビュー] The Numerics of GANs

Lars Mescheder, Sebastian Nowozin+1 more|arXiv (Cornell University)|May 30, 2017

Generative Adversarial Networks and Image Synthesis参考文献 22被引用数 188

ひとこと要約

本論文は、勾配ジャコビ行列の固有値が原因で GAN の学習収束がしばしば崩れる理由を分析し、さまざまなアーキテクチャや発散度を横断して学習を安定化させるための Consensus Optimization を提案します。

ABSTRACT

In this paper, we analyze the numerics of common algorithms for training Generative Adversarial Networks (GANs). Using the formalism of smooth two-player games we analyze the associated gradient vector field of GAN training objectives. Our findings suggest that the convergence of current algorithms suffers due to two factors: i) presence of eigenvalues of the Jacobian of the gradient vector field with zero real-part, and ii) eigenvalues with big imaginary part. Using these findings, we design a new algorithm that overcomes some of these limitations and has better convergence properties. Experimentally, we demonstrate its superiority on training common GAN architectures and show convergence on GAN architectures that are known to be notoriously hard to train.

研究の動機と目的

• Identify why simultaneous gradient ascent struggles to find local Nash equilibria in GANs.
• Model the GAN training as a smooth two-player game and study the gradient vector field.
• Propose a robust optimization method to improve convergence and stability.
• Empirically validate the method on common GAN architectures and divergences.

提案手法

• Frame GAN training as a smooth two-player game with a gradient vector field v(x).
• Analyze the Jacobian of v(x) to identify causes of non-convergence, including zero-real-part eigenvalues and large imaginary parts.
• Introduce a modified vector field w(x)=v(x) - γ ∇L(x) with L(x)=½||v(x)||² to obtain better convergence properties.
• Derive Consensus Optimization (Algorithm 2) where the modified game uses tilde f and tilde g with a regularizer L.
• Provide convergence results showing local convergence to a local Nash equilibrium under certain conditions (negative semi-definite v′(x) and appropriate γ, h).
• Demonstrate that the eigenvalue spectrum is shifted left, improving stability, and relate this to second-order/implicit-Euler interpretations.

実験結果

リサーチクエスチョン

• RQ1 What specifically causes simultaneous gradient ascent to fail to converge to local Nash equilibria in GANs?
• RQ2 How can the gradient dynamics be adapted to ensure robust convergence in two-player GAN games?
• RQ3 Does the proposed consensus optimization approach improve stability and convergence across different GAN architectures and divergence measures?

主な発見

• Eigenvalues of the Jacobian with zero real part and large imaginary parts hinder convergence of SimGA in GANs.
• Consensus Optimization moves eigenvalues to the left in the complex plane, improving stability and allowing reasonable step sizes.
• The method yields stable training on CIFAR-10 and CelebA with architectures known to be hard to train by standard methods.
• Training with Consensus Optimization gives more stable generator/discriminator losses and competitive inception scores compared with alternating gradient ascent.
• Consensus optimization is compatible with various GAN architectures and divergence measures, functioning as a numerically robust alternative to standard methods.