[Paper Review] Prescribed Generative Adversarial Networks

Adji Bousso Dieng, Francisco J. R. Ruiz+2 more|arXiv (Cornell University)|Oct 9, 2019

Generative Adversarial Networks and Image Synthesis67 references47 citations

TL;DR

PresGANs add noise to the generator output and use entropy-regularized adversarial loss to prevent mode collapse and enable tractable predictive log-likelihood evaluation for GANs.

ABSTRACT

Generative adversarial networks (GANs) are a powerful approach to unsupervised learning. They have achieved state-of-the-art performance in the image domain. However, GANs are limited in two ways. They often learn distributions with low support---a phenomenon known as mode collapse---and they do not guarantee the existence of a probability density, which makes evaluating generalization using predictive log-likelihood impossible. In this paper, we develop the prescribed GAN (PresGAN) to address these shortcomings. PresGANs add noise to the output of a density network and optimize an entropy-regularized adversarial loss. The added noise renders tractable approximations of the predictive log-likelihood and stabilizes the training procedure. The entropy regularizer encourages PresGANs to capture all the modes of the data distribution. Fitting PresGANs involves computing the intractable gradients of the entropy regularization term; PresGANs sidestep this intractability using unbiased stochastic estimates. We evaluate PresGANs on several datasets and found they mitigate mode collapse and generate samples with high perceptual quality. We further found that PresGANs reduce the gap in performance in terms of predictive log-likelihood between traditional GANs and variational autoencoders (VAEs).

Motivation & Objective

• Address mode collapse in GANs by promoting high-entropy generator distributions through entropy regularization.
• Enable tractable estimation of predictive log-likelihood for GANs by defining a well-defined generative density.
• Develop unbiased stochastic gradient estimators for entropy gradients to fit PresGANs effectively.
• Demonstrate empirical improvements in mode coverage and sample quality across datasets.

Proposed method

• Define the generative process p_theta(x) as an infinite mixture with Gaussian z and Gaussian x|z to guarantee a well-defined density.
• Introduce an entropy regularizer into the GAN objective: L_PresGAN = L_GAN - lambda * H(p_theta(x)).
• Add noise to the generator output by perturbing x with sigma, ensuring existence of p_theta(x) and enabling entropy computation.
• Compute unbiased Monte Carlo estimates of the entropy gradient using reparameterization and Hamiltonian Monte Carlo (HMC) for z samples.
• Formulate unbiased gradient estimates for theta, including a term for the entropy gradient, via reparameterization and Monte Carlo averages.
• Update the discriminator φ as in standard GANs while applying consistent data-noising to real samples to stabilize training.

Experimental results

Research questions

• RQ1 Can entropy regularization via additive output noise prevent mode collapse in GANs?
• RQ2 Is it possible to obtain tractable predictive log-likelihoods for GANs by replacing the implicit generator with a well-defined density?
• RQ3 How can one effectively estimate the gradients of the entropy term in PresGANs to train the model?
• RQ4 Do PresGANs improve mode coverage and sample diversity without sacrificing perceptual quality?

Key findings

• PresGANs mitigate mode collapse and recover multiple data modes in synthetic and real-data experiments.
• Generated samples achieve high perceptual quality while maintaining diverse coverage of the data distribution.
• Predictive log-likelihood gaps between GANs and VAEs are reduced when using PresGANs, indicating improved generalization assessment.
• The entropy-regularized objective enables tractable approximations of log-likelihood via importance sampling.
• Unbiased gradient estimates for the entropy term can be obtained using reparameterization and HMC-based posterior samples.
• Training stability is aided by data-noising of real data and controlled variance in the generator.”