[Paper Review] On distinguishability criteria for estimating generative models

Ian Goodfellow|arXiv (Cornell University)|Dec 19, 2014

Generative Adversarial Networks and Image Synthesis2 references87 citations

TL;DR

This paper analyzes the theoretical relationship between noise-contrastive estimation (NCE), generative adversarial networks (GANs), and maximum likelihood estimation (MLE). It shows that a dynamic-generator variant of NCE is equivalent to MLE, but GANs cannot recover MLE gradients without abandoning the distinguishability game, and existing theory does not guarantee GAN convergence in non-convex settings.

ABSTRACT

Two recently introduced criteria for estimation of generative models are both based on a reduction to binary classification. Noise-contrastive estimation (NCE) is an estimation procedure in which a generative model is trained to be able to distinguish data samples from noise samples. Generative adversarial networks (GANs) are pairs of generator and discriminator networks, with the generator network learning to generate samples by attempting to fool the discriminator network into believing its samples are real data. Both estimation procedures use the same function to drive learning, which naturally raises questions about how they are related to each other, as well as whether this function is related to maximum likelihood estimation (MLE). NCE corresponds to training an internal data model belonging to the {\em discriminator} network but using a fixed generator network. We show that a variant of NCE, with a dynamic generator network, is equivalent to maximum likelihood estimation. Since pairing a learned discriminator with an appropriate dynamically selected generator recovers MLE, one might expect the reverse to hold for pairing a learned generator with a certain discriminator. However, we show that recovering MLE for a learned generator requires departing from the distinguishability game. Specifically: (i) The expected gradient of the NCE discriminator can be made to match the expected gradient of MLE, if one is allowed to use a non-stationary noise distribution for NCE, (ii) No choice of discriminator network can make the expected gradient for the GAN generator match that of MLE, and (iii) The existing theory does not guarantee that GANs will converge in the non-convex case. This suggests that the key next step in GAN research is to determine whether GANs converge, and if not, to modify their training algorithm to force convergence.

Motivation & Objective

• To clarify the theoretical relationship between NCE, GANs, and MLE in the context of generative modeling.
• To investigate whether GANs can recover the gradient updates of maximum likelihood estimation.
• To examine the convergence properties of GANs under the distinguishability game framework.
• To identify why GANs may underfit in practice despite theoretical consistency.
• To assess whether the distinguishability game function can be used to achieve MLE via adversarial training.

Proposed method

• Compares the distinguishability game value function used in both NCE and GANs, defined as $ V(p_c, p_g) = \mathbb{E}_{\mathbf{x} \sim p_d} \log p_c(y=1|\mathbf{x}) + \mathbb{E}_{\mathbf{x} \sim p_g} \log p_c(y=0|\mathbf{x}) $.
• Analyzes NCE with a fixed noise distribution and shows that a dynamic generator version of NCE is equivalent to MLE.
• Derives the expected gradient of the generator in GANs and compares it to the MLE gradient, showing a fundamental mismatch.
• Uses the logistic sigmoid function to model the discriminator output $ p_c(y=1|\mathbf{x}) = \sigma(a(\mathbf{x})) $, and derives the generator cost function.
• Demonstrates that the MLE gradient requires $ f(\mathbf{x}) = -\frac{p_d(\mathbf{x})}{p_g(\mathbf{x})} $, while the GAN cost uses $ f(\mathbf{x}) = -\zeta(a(\mathbf{x})) $, where $ \zeta $ is the softplus function.
• Identifies that high variance in gradient estimates for MLE arises when the discriminator is not confident, making training unstable without additional mechanisms.

Experimental results

Research questions

• RQ1 Can a modified version of NCE with a dynamic generator recover maximum likelihood estimation?
• RQ2 Is there a way to make GANs produce gradients equivalent to those of maximum likelihood estimation?
• RQ3 Why do GANs often fail to converge in practice despite theoretical consistency?
• RQ4 What is the relationship between the distinguishability game and maximum likelihood estimation?
• RQ5 Can the distinguishability game be modified to ensure convergence in non-convex settings?

Key findings

• A variant of NCE with a dynamic generator is mathematically equivalent to maximum likelihood estimation.
• The expected gradient of the GAN generator cannot match the MLE gradient under any choice of discriminator network.
• The distinguishability game cost function in GANs leads to a different gradient than MLE, specifically using the softplus function instead of the exponential function required for MLE.
• The high variance in MLE gradient estimates under the distinguishability game framework arises because the discriminator only gives significant gradients when it is highly confident, which is rare for untrained generators.
• Non-convergence of gradient-based learning in non-convex games is a plausible explanation for the underfitting observed in GANs, though no positive convergence guarantee exists in the current theoretical framework.
• The paper suggests that future work should focus on ensuring convergence in GAN training, possibly by modifying the training algorithm to enforce equilibrium computation.