[Paper Review] Good Semi-supervised Learning that Requires a Bad GAN

Zihang Dai, Zhilin Yang+3 more|arXiv (Cornell University)|May 27, 2017

Generative Adversarial Networks and Image Synthesis18 references231 citations

TL;DR

The paper shows that for GAN-based semi-supervised learning, a bad (complement) generator is beneficial, and provides a practical method that outperforms feature matching by encouraging high-entropy, low-density complement samples guided by a KL-divergence objective and conditional entropy in the discriminator.

ABSTRACT

Semi-supervised learning methods based on generative adversarial networks (GANs) obtained strong empirical results, but it is not clear 1) how the discriminator benefits from joint training with a generator, and 2) why good semi-supervised classification performance and a good generator cannot be obtained at the same time. Theoretically, we show that given the discriminator objective, good semisupervised learning indeed requires a bad generator, and propose the definition of a preferred generator. Empirically, we derive a novel formulation based on our analysis that substantially improves over feature matching GANs, obtaining state-of-the-art results on multiple benchmark datasets.

Motivation & Objective

• Motivate semi-supervised learning with GANs and understand why a good generator and strong SSL performance may not coincide.
• Theoretically characterize when a generator contributes to SSL under the standard K+1 discriminator objective.
• Define and motivate a complement generator that generates samples in feature-space regions that aid decision boundary placement.
• Propose a practical SSL method that enforces high generator entropy and low-density sample generation, improving over feature matching.

Proposed method

• Analyze the GAN-based SSL objective with a K+1 classifier in the discriminator and show that good SSL requires a complement (bad) generator.
• Define a complement generator in the feature space and establish conditions under which the discriminator learns correct, high-density boundaries (Proposition 2).
• Introduce a generator objective that combines generator entropy maximization (via variational encoder or pull-away term) with a low-density sample penalty (using a PixelCNN++ density model) and a KL-divergence target aligning p_G with a complement distribution in input space (Eq. 4).
• Add a conditional entropy term to the discriminator objective to enforce strong true-fake belief on unlabeled data (Eq. 5).
• Train generator and discriminator iteratively, with entropy-increasing methods and low-density enforcement, to approximate the complement-generator framework.

Experimental results

Research questions

• RQ1 Does the standard K+1 discriminator objective in GAN-based SSL benefit from a generator, and under what conditions is the generator beneficial or detrimental?
• RQ2 Can a complement (bad) generator, rather than a perfect one, improve semi-supervised classification performance, and why?
• RQ3 How can we design a practical objective for the generator and discriminator to realize the complement-generator idea in high-dimensional data?
• RQ4 What empirical gains do complement-generator-inspired methods achieve on standard SSL benchmarks compared to feature matching?

Key findings

• A perfect generator does not improve generalization under the K+1 discriminator formulation (Proposition 1).
• A complement (bad) generator helps place decision boundaries in low-density feature-space regions (Proposition 2).
• The proposed generator objective, combining entropy maximization and low-density sampling toward a complement distribution, improves over feature matching on MNIST, SVHN, and CIFAR-10 when using the same discriminator architecture.
• Adding a conditional entropy term to the discriminator enforces strong true-fake belief on unlabeled data, aiding the SSL process.
• Empirically, the method achieves state-of-the-art results on MNIST, SVHN, and CIFAR-10 with small discriminators, and is competitive with larger models and self-ensembling approaches.