[Paper Review] Large Scale Adversarial Representation Learning

Jeff Donahue, Karen Simonyan|arXiv (Cornell University)|Jul 4, 2019

Generative Adversarial Networks and Image Synthesis36 references196 citations

TL;DR

BigBiGAN extends BigGAN with an encoder and a joint discriminator to enable adversarial representation learning, achieving state-of-the-art unsupervised ImageNet representations and improving unconditional image generation.

ABSTRACT

Adversarially trained generative models (GANs) have recently achieved compelling image synthesis results. But despite early successes in using GANs for unsupervised representation learning, they have since been superseded by approaches based on self-supervision. In this work we show that progress in image generation quality translates to substantially improved representation learning performance. Our approach, BigBiGAN, builds upon the state-of-the-art BigGAN model, extending it to representation learning by adding an encoder and modifying the discriminator. We extensively evaluate the representation learning and generation capabilities of these BigBiGAN models, demonstrating that these generation-based models achieve the state of the art in unsupervised representation learning on ImageNet, as well as in unconditional image generation. Pretrained BigBiGAN models -- including image generators and encoders -- are available on TensorFlow Hub (https://tfhub.dev/s?publisher=deepmind&q=bigbigan).

Motivation & Objective

• Motivate representation learning from high-quality generative models rather than self-supervised tasks.
• Develop BigBiGAN by integrating an encoder and a joint discriminator with a BigGAN-style generator.
• Systematically evaluate representation learning performance on ImageNet via linear probing and assess generation metrics (IS, FID).
• Investigate design choices (encoder architecture, unary discriminator terms, generator capacity) and their impact on both representation and generation.

Proposed method

• Use a BiGAN/ALI framework with a BigGAN-style generator G and an encoder E to map data x to latent z.
• Introduce a joint discriminator D that scores data x, latent z, and their pair (x,z) through unary and joint terms.
• Adopt a discriminator architecture with F (for x), H (for z), and J (joint x,z) to compute scores s_x, s_z, s_xz.
• Optimize encoder-generator losses to fool D while D learns to distinguish encoder-induced pairs from generator-induced pairs.
• Train with an asymmetric E/G setup, higher-resolution E inputs, and decoupled E/G optimization to accelerate convergence.
• Compare variants through ablations including non-deterministic E, unary loss terms, and generator capacity; evaluate via ImageNet linear probing and generation metrics.

Experimental results

Research questions

• RQ1 Can a BigGAN-based generator paired with an encoder in a BiGAN/ALI framework learn high-quality unsupervised representations for ImageNet?
• RQ2 Do unary discriminator terms and a stable joint discriminator improve representation learning without sacrificing generation quality?
• RQ3 How do encoder resolution, generator capacity, and decoupled E/G optimization affect downstream classification and unconditional image generation?
• RQ4 Is the learned representation from BigBiGAN competitive with state-of-the-art self-supervised methods on ImageNet?
• RQ5 What is the impact of representation learning on unconditional generation metrics (IS, FID) for BigBiGAN?

Key findings

• BigBiGAN matches or exceeds state-of-the-art unsupervised ImageNet results among recent methods using generative models.
• Adding unary terms to the discriminator and increasing encoder capacity improves both representation quality (Cls.) and generation metrics (IS, FID).
• Non-deterministic encoder (sampling z|x) yields better downstream classification than deterministic encodings.
• Increasing E resolution and G capacity generally improves representation learning, though training efficiency decreases; higher-resolution E can boost generation quality (FID).
• Decoupling E and G optimization (higher E learning rate) accelerates training and improves linear classification performance by several percent.
• Unsupervised BigBiGAN generation with high-resolution E can outperform prior unsupervised methods on IS/FID benchmarks and approaches or matches supervised baselines on some configurations.