[Paper Review] Training Generative Adversarial Networks with Limited Data

Tero Karras, Miika Aittala+4 more|arXiv (Cornell University)|Jun 11, 2020

Generative Adversarial Networks and Image Synthesis36 references932 citations

TL;DR

Introduces adaptive discriminator augmentation (ADA) to stabilize GAN training on small datasets without changing losses or architectures, achieving high-quality results with thousands of images and new CIFAR-10 benchmarks.

ABSTRACT

Training generative adversarial networks (GAN) using too little data typically leads to discriminator overfitting, causing training to diverge. We propose an adaptive discriminator augmentation mechanism that significantly stabilizes training in limited data regimes. The approach does not require changes to loss functions or network architectures, and is applicable both when training from scratch and when fine-tuning an existing GAN on another dataset. We demonstrate, on several datasets, that good results are now possible using only a few thousand training images, often matching StyleGAN2 results with an order of magnitude fewer images. We expect this to open up new application domains for GANs. We also find that the widely used CIFAR-10 is, in fact, a limited data benchmark, and improve the record FID from 5.59 to 2.42.

Motivation & Objective

• Motivate the problem of GAN overfitting when data is scarce and quantify its impact on training stability.
• Propose an augmentation-based solution that prevents discriminator overfitting without leaking to generated images.
• Develop an adaptive mechanism to control augmentation strength during training.
• Demonstrate effectiveness across multiple datasets and transfer learning scenarios, including CIFAR-10 as a limited data benchmark.
• Compare with alternative methods and establish practical guidelines for small-data GANs.

Proposed method

• Propose stochastic discriminator augmentation that applies a diverse set of differentiable augmentations to all images shown to the discriminator, including during generator training.
• Develop non-leaking augmentation theory to ensure augmentations do not leak into generated images, with probabilistic application controlled by p.
• Design an augmentation pipeline with 18 transformations across six categories, applied with probability p in a fixed order, and ensure differentiability for backpropagation.
• Introduce adaptive discriminator augmentation (ADA) that tunes augmentation strength p based on overfitting heuristics using discriminator outputs on training/validation/generated data.
• Use two overfitting heuristics r_v and r_t to guide p, ultimately adopting r_t with a target value (0.6) to drive adaptivity.
• Evaluate ADA in from-scratch and transfer-learning GAN setups, comparing with alternatives like bCR and various architecture/discriminator tweaks.

Experimental results

Research questions

• RQ1 Can discriminator overfitting in GANs trained on limited data be mitigated without changing loss functions or network architectures?
• RQ2 Do non-leaking, stochastic discriminator augmentations stabilize GAN training across different dataset sizes and domains?
• RQ3 How should augmentation strength be adaptively tuned during training to prevent overfitting while preserving generator quality?
• RQ4 Does ADA improve performance on small datasets and transfer learning scenarios, and how does it compare to related methods like bCR?
• RQ5 Is CIFAR-10 a truly limited-data benchmark, and can ADA substantially improve its metrics?

Key findings

• ADA substantially stabilizes GAN training on limited data, preventing discriminator overfitting and divergence.
• A diverse set of non-leaking, differentiable augmentations can be applied to all discriminator inputs (and generator inputs) without leaking to generated images.
• Adaptive control of augmentation strength (p) via overfitting heuristics yields better FID/KID scores than fixed augmentation, across dataset sizes.
• ADA achieves strong results from scratch and via transfer learning, often matching or exceeding StyleGAN2 quality with an order of magnitude fewer images.
• On CIFAR-10, ADA improves FID from 5.59 to 2.42 and IS in class-conditional settings, indicating CIFAR-10 is a limited data benchmark and ADA provides substantial gains.
• ADA remains energy-efficient and does not significantly increase training cost.