[Paper Review] High-Fidelity Image Generation With Fewer Labels

Mario Lučić, Michael Tschannen+4 more|arXiv (Cornell University)|Mar 6, 2019

Image Processing Techniques and ApplicationsEngineering42 references95 citations

TL;DR

The paper shows that self- and semi-supervised learning can enable high-fidelity ImageNet image generation with far fewer labels, matching or surpassing BigGAN using as little as 10% of the labels and 20% achieving state-of-the-art FID/IS.

ABSTRACT

Deep generative models are becoming a cornerstone of modern machine learning. Recent work on conditional generative adversarial networks has shown that learning complex, high-dimensional distributions over natural images is within reach. While the latest models are able to generate high-fidelity, diverse natural images at high resolution, they rely on a vast quantity of labeled data. In this work we demonstrate how one can benefit from recent work on self- and semi-supervised learning to outperform the state of the art on both unsupervised ImageNet synthesis, as well as in the conditional setting. In particular, the proposed approach is able to match the sample quality (as measured by FID) of the current state-of-the-art conditional model BigGAN on ImageNet using only 10% of the labels and outperform it using 20% of the labels.

Motivation & Objective

• Demonstrate how self-supervised learning can provide semantic features to guide GAN training.
• Investigate semi-supervised labeling to infer conditional information for GANs with limited ground-truth labels.
• Evaluate pre-trained and co-training approaches to reduce labeled data while maintaining high sample quality.
• Assess self-supervision during GAN training as a stabilizing factor for high-fidelity image synthesis.

Proposed method

• Use a conditional BigGAN architecture as the backbone for high-fidelity ImageNet generation.
• Explore three families of label-reduction approaches: (i) pre-trained self-/semi-supervised methods with clustering or linear classifiers, (ii) co-training where an auxiliary classifier predicts labels for unlabeled data during GAN training, and (iii) self-supervision during GAN training added to discriminators.
• Incorporate rotation-based self-supervision as an auxiliary task to the discriminator and generator losses.
• Experiment with hard vs soft labels in the projection discriminator and study stability across 5%, 10%, and 20% labeled data.
• Evaluate on ImageNet at 128x128 and 256x256 resolutions using FID and Inception Score (IS).
• Open-source code is provided for reproducibility.

Experimental results

Research questions

• RQ1 Can self-supervised representations enable high-fidelity ImageNet generation with limited labels?
• RQ2 How do pre-training, co-training, and self-supervised objectives compare in reducing labeled data while preserving or improving sample quality?
• RQ3 Does self-supervision during GAN training stabilize high-resolution GANs trained with limited labels?

Key findings

• Clustering with a self-supervised representation yields a new state-of-the-art in unsupervised ImageNet generation, reducing FID and increasing IS relative to basic unsupervised setups.
• Pre-trained semi-supervised S2 GAN matches BigGAN with 20% labeled data and closely approaches it with 10% labels.
• Co-training S2 GAN-CO outperforming unsupervised baselines across label percentages, with 20% labels achieving FID 13.9 and IS 49.2, close to supervised baselines.
• Self-supervision during GAN training (S3 GAN) enables matching BigGAN with 10% labels and surpassing it with 20% labels, improving FID by roughly 5–10% in several settings.
• Hard labels (predicted) outperform soft labels in several configurations, and self-supervision consistently improves stability and sample quality across methods.