[Paper Review] Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks

Jost Tobias Springenberg|arXiv (Cornell University)|Nov 19, 2015

Digital Media Forensic DetectionComputer Science397 citations

TL;DR

This paper introduces Categorical Generative Adversarial Networks (CatGAN), a method for unsupervised and semi-supervised image classification that jointly trains a discriminative classifier and an adversarial generator. By maximizing mutual information between inputs and predicted class distributions while enforcing robustness to adversarial examples, CatGAN achieves state-of-the-art performance on CIFAR-10 with 19.58% error using only 400 labeled examples per class, while generating high-fidelity images.

ABSTRACT

In this paper we present a method for learning a discriminative classifier from unlabeled or partially labeled data. Our approach is based on an objective function that trades-off mutual information between observed examples and their predicted categorical class distribution, against robustness of the classifier to an adversarial generative model. The resulting algorithm can either be interpreted as a natural generalization of the generative adversarial networks (GAN) framework or as an extension of the regularized information maximization (RIM) framework to robust classification against an optimal adversary. We empirically evaluate our method - which we dub categorical generative adversarial networks (or CatGAN) - on synthetic data as well as on challenging image classification tasks, demonstrating the robustness of the learned classifiers. We further qualitatively assess the fidelity of samples generated by the adversarial generator that is learned alongside the discriminative classifier, and identify links between the CatGAN objective and discriminative clustering algorithms (such as RIM).

Motivation & Objective

• To develop a unified framework for unsupervised and semi-supervised learning that combines generative modeling with discriminative classification.
• To improve generalization of deep neural network classifiers by enforcing robustness to adversarial examples generated during training.
• To enable effective learning from limited labeled data by leveraging unlabeled data through mutual information maximization.
• To explore the connection between adversarial training and discriminative clustering methods such as Regularized Information Maximization (RIM).
• To evaluate the fidelity of generated samples and the classifier's performance on standard benchmarks like MNIST and CIFAR-10.

Proposed method

• The method introduces a novel objective function that trades-off mutual information between input data and predicted class distributions against classifier robustness to adversarial examples.
• It extends the GAN framework by training a discriminative classifier D to predict categorical class probabilities, while a generator G produces adversarial examples to challenge D.
• The classifier is optimized to maximize mutual information I(X; Y|D) between inputs X and predicted labels Y, promoting disentangled, informative representations.
• The generator is trained to produce realistic-looking examples that can fool the classifier, thereby regularizing D against overfitting to spurious features.
• The framework supports both unsupervised (no labels) and semi-supervised (few labeled examples) learning by leveraging unlabeled data to improve generalization.
• The method is empirically validated on synthetic data, MNIST, and CIFAR-10, with ablation studies on the impact of adversarial regularization and label efficiency.

Experimental results

Research questions

• RQ1 Can adversarial training improve the robustness and generalization of deep neural network classifiers in semi-supervised learning?
• RQ2 How does maximizing mutual information between inputs and predicted class distributions enhance representation learning in the absence of full supervision?
• RQ3 To what extent can a jointly trained generator produce high-fidelity samples that reflect the underlying data distribution?
• RQ4 How does the CatGAN objective relate to existing discriminative clustering methods such as Regularized Information Maximization (RIM)?
• RQ5 What is the performance of CatGAN on standard image classification benchmarks when only a small number of labeled examples are available?

Key findings

• On CIFAR-10 with 400 labeled examples per class, CatGAN achieved a test error of 19.58% (±0.58), outperforming several baselines including Conv-Ladder and standard GANs.
• With full supervision, CatGAN achieved a test error of 23.4% (±0.2), comparable to state-of-the-art methods like Conv-CatGAN and Conv-Ladder.
• The unsupervised CatGAN model generated high-fidelity images on MNIST, CIFAR-10, and LFW, as confirmed by qualitative visual inspection in Figure 3.
• The model achieved a log-likelihood of 237 ± 6 on MNIST, comparable to the 225 ± 2 reported by Goodfellow et al. (2014) for standard GANs, though log-likelihood comparisons are cautioned against due to estimation bias.
• The method was found to be equivalent to Regularized Information Maximization (RIM) when the generator was removed and L2 regularization was applied, confirming theoretical connections to discriminative clustering.
• The adversarial generator significantly improved classifier robustness, with no evidence that L2 regularization further improved performance in the authors' experiments.