[Paper Review] Smoothness and Stability in GANs

Casey Chu, Kentaro Minami+1 more|arXiv (Cornell University)|Feb 11, 2020

Model Reduction and Neural Networks45 references20 citations

TL;DR

This paper develops a theoretical framework linking smoothness of the GAN loss function to training stability, showing that gradient descent on the generator converges to stationarity only if the loss and generator architecture satisfy specific smoothness conditions. It derives conditions for stability, explains why common GAN techniques like gradient penalties and spectral normalization are effective, and proposes a novel GAN using reproducing kernel Hilbert spaces that satisfies all smoothness requirements, achieving improved stability.

ABSTRACT

Generative adversarial networks, or GANs, commonly display unstable behavior during training. In this work, we develop a principled theoretical framework for understanding the stability of various types of GANs. In particular, we derive conditions that guarantee eventual stationarity of the generator when it is trained with gradient descent, conditions that must be satisfied by the divergence that is minimized by the GAN and the generator's architecture. We find that existing GAN variants satisfy some, but not all, of these conditions. Using tools from convex analysis, optimal transport, and reproducing kernels, we construct a GAN that fulfills these conditions simultaneously. In the process, we explain and clarify the need for various existing GAN stabilization techniques, including Lipschitz constraints, gradient penalties, and smooth activation functions.

Motivation & Objective

• To resolve the theoretical mystery behind GAN training instability despite widespread use of regularization techniques.
• To identify precise smoothness conditions on the GAN loss and generator architecture that guarantee convergence to stationarity under gradient descent.
• To unify and explain the effectiveness of existing stabilization techniques—like gradient penalties and spectral normalization—within a single theoretical framework.
• To construct a new GAN variant that satisfies all derived smoothness conditions, ensuring stable training.

Proposed method

• Formalize GAN training as gradient descent on the generator’s objective function $ J(\mu_\theta) = \sup_\varphi \mathcal{J}(\mu_\theta, \varphi) $, reducing the minimax problem to a single optimization.
• Apply tools from convex analysis and optimal transport to derive sufficient conditions for $ J(\mu_\theta) $ to be $ L $-smooth, ensuring gradient descent stability.
• Use reproducing kernel Hilbert spaces (RKHS) with Gaussian kernels to construct a loss function that satisfies all smoothness conditions, enabling theoretical convergence guarantees.
• Introduce a regularized GAN objective where the generator’s output distribution is penalized via an RKHS norm, enforcing smoothness in the generator's parameterization.
• Derive the convex conjugate of the RKHS norm penalty to enable tractable optimization and theoretical analysis.
• Propose a dimension-free kernel $ K(x,y) = e^{-\pi \|x-y\|^2} $ for numerical stability and theoretical tractability.

Experimental results

Research questions

• RQ1 What smoothness conditions on the GAN loss and generator architecture are necessary and sufficient for gradient descent to converge to a stationary point?
• RQ2 Why do common GAN regularization techniques—such as gradient penalties and spectral normalization—improve training stability, despite overlapping objectives?
• RQ3 Can a GAN be constructed that provably satisfies all smoothness conditions, ensuring stable training?
• RQ4 How do existing GAN variants like WGAN and GAN with spectral normalization relate to the proposed smoothness framework?

Key findings

• Most standard GAN losses, including WGAN and standard GAN, fail to satisfy the full set of smoothness conditions required for stable gradient descent, explaining their empirical instability.
• The proposed GAN with an RKHS-based penalty achieves full smoothness, guaranteeing that gradient descent on the generator converges to a stationary point.
• Gradient penalties and spectral normalization are shown to be effective because they enforce the smoothness conditions required for stability, providing a unified theoretical explanation.
• The use of smooth activation functions is theoretically justified as they help maintain the smoothness of the generator’s output distribution.
• Simultaneous application of spectral normalization and gradient penalties is shown to be more effective than either alone, as they jointly enforce different components of the smoothness condition.
• A critical parameter $ \sigma = (2\pi)^{-1/2} $ is recommended for the Gaussian kernel to ensure numerical stability and theoretical consistency.