[Paper Review] Adversarial Feature Learning

Jeff Donahue, Philipp Krähenbühl+1 more|arXiv (Cornell University)|May 31, 2016

Generative Adversarial Networks and Image Synthesis31 references721 citations

TL;DR

BiGANs learn an inverse mapping to GANs by training an encoder alongside a generator and a discriminator, yielding useful unsupervised features for downstream tasks.

ABSTRACT

The ability of the Generative Adversarial Networks (GANs) framework to learn generative models mapping from simple latent distributions to arbitrarily complex data distributions has been demonstrated empirically, with compelling results showing that the latent space of such generators captures semantic variation in the data distribution. Intuitively, models trained to predict these semantic latent representations given data may serve as useful feature representations for auxiliary problems where semantics are relevant. However, in their existing form, GANs have no means of learning the inverse mapping -- projecting data back into the latent space. We propose Bidirectional Generative Adversarial Networks (BiGANs) as a means of learning this inverse mapping, and demonstrate that the resulting learned feature representation is useful for auxiliary supervised discrimination tasks, competitive with contemporary approaches to unsupervised and self-supervised feature learning.

Motivation & Objective

• Motivate unsupervised feature learning via GANs by learning the inverse mapping from data to latent space.
• Introduce Bidirectional Generative Adversarial Networks (BiGANs) with an encoder that maps data to latent codes.
• Theoretically analyze optimal discriminators, generators, and encoders and show inversion properties.
• Empirically evaluate BiGANs on MNIST and ImageNet to assess transferability of learned features to supervised tasks.

Proposed method

• Extend GANs by adding an encoder E that maps data x to latent z.
• Modify the discriminator to assess pairs (x, z) from real data versus (G(z), z) from generated data.
• Optimize a minimax objective V(D,E,G) that maximizes over D and minimizes over E,G (Equation 3).
• Prove that at optimum, P_EX equals P_GZ and that E and G invert each other almost everywhere (Theorems 1 and 2).
• Show BiGANs correspond to an 0-1 style autoencoder loss (Theorem 3) in the optimal setting.
• Generalize BiGAN to handle different input/output spaces via g_X and g_Z (Section 3.5).
• Train BiGANs with standard alternating gradient methods and practical “inverse objective” for stronger gradients (Section 3.4).

Experimental results

Research questions

• RQ1 CanBiGANs learn meaningful inverse mappings to GANs by jointly training an encoder with a generator and discriminator?
• RQ2 Do BiGANs produce latent representations that are useful for downstream supervised tasks without labeled data?
• RQ3 What are the theoretical properties of BiGANs regarding optimality and inversion of encoder and generator?
• RQ4 How do BiGANs compare to other unsupervised/self-supervised feature learning methods on real-world image datasets?
• RQ5 How does the BiGAN framework extend to higher-resolution inputs and different feature spaces?

Key findings

• The BiGAN objective yields a Jensen–Shannon divergence between joint distributions P_EX and P_GZ, with the global optimum at P_EX = P_GZ.
• At optimum, the encoder and generator invert each other almost everywhere (G(E(x)) = x and E(G(z)) = z for data and latent supports).
• BiGAN encoders learn features that function as latent representations for semantic attributes, acting similarly to an ℓ0 autoencoder in purpose but without assuming data structure.
• On permutation-invariant MNIST, BiGAN features achieve competitive 1NN accuracy compared to baselines like latent regressor and autoencoders (97.39% vs 97.30–97.63% across variants).
• On ImageNet, BiGANs yield competitive transfer performance when used as pretrained feature extractors, with qualitative gains in learned filters and reconstructions (Figure 3 & 4).
• BiGAN representations transfer to PASCAL VOC tasks (classification/detection/segmentation) in line with contemporary unsupervised/self-supervised methods (Table 3).