[Paper Review] Stabilizing Training of Generative Adversarial Networks through Regularization

Kevin A. Roth, Aurélien Lucchi+2 more|arXiv (Cornell University)|May 25, 2017

Generative Adversarial Networks and Image SynthesisComputer Science27 references204 citations

TL;DR

The paper introduces a noise-induced regularizer for discriminators in f-GANs to stabilize GAN training, addressing dimensional misspecification and improving sample quality across architectures.

ABSTRACT

Deep generative models based on Generative Adversarial Networks (GANs) have demonstrated impressive sample quality but in order to work they require a careful choice of architecture, parameter initialization, and selection of hyper-parameters. This fragility is in part due to a dimensional mismatch or non-overlapping support between the model distribution and the data distribution, causing their density ratio and the associated f-divergence to be undefined. We overcome this fundamental limitation and propose a new regularization approach with low computational cost that yields a stable GAN training procedure. We demonstrate the effectiveness of this regularizer across several architectures trained on common benchmark image generation tasks. Our regularization turns GAN models into reliable building blocks for deep learning.

Motivation & Objective

• Motivate and address instability in GAN training caused by dimensional mismatch between data and model distributions.
• Introduce a noise-based regularization framework that yields a well-defined, robust f-GAN objective.
• Provide an efficient, implementable regularizer that stabilizes training across various GAN architectures and datasets.
• Evaluate stability and sample quality across multiple models and benchmarks.

Proposed method

• Derive a noise-convolution regularization for f-GANs by convolving data with Gaussian noise to obtain a smoother discriminant.
• Approximate the resulting regularized objective with a gradient-norm penalty that weights the squared gradient by the second derivative of the f-composite function.
• Specialize the regularizer for Jensen-Shannon GANs, yielding a gradient-norm penalty on the discriminator and its logit, computable via backpropagation.
• Propose an annealing scheme for the regularization strength gamma to balance early stability with later performance.
• Provide Algorithm 1 detailing Regularized JS-GAN with discriminator gradient penalty and generator updates.

Experimental results

Research questions

• RQ1 Can adding a noise-induced regularizer to the discriminator stabilize training when data and model distributions have non-overlapping supports or dimensional mismatch?
• RQ2 How does the proposed regularizer affect stability, convergence, and sample quality across diverse GAN architectures and datasets?
• RQ3 What is the effect of annealing the regularization strength on training dynamics and final generated sample quality?
• RQ4 How does regularization compare to explicit input noise in terms of stability and sample quality?
• RQ5 Does cross-testing reveal generalization benefits of regularized GANs over unregularized ones?

Key findings

• The regularizer stabilizes GAN training across a range of architectures and datasets, including CelebA, CIFAR-10, and LSUN bedrooms.
• Regularized GANs can be trained essentially indefinitely without collapse in challenging dimensional-misspecification scenarios.
• Annealing the regularization strength gamma improves stability early in training and remains beneficial when gamma is reduced but nonzero.
• Regularization yields higher visual quality samples than unregularized models and outperforms explicit input noise in stability and quality across experiments.
• Cross-testing shows the regularized model’s discriminator generalizes better to distinguishing its own generated data, indicating improved generalization properties.