[Paper Review] Differentiable Augmentation for Data-Efficient GAN Training

Shengyu Zhao, Zhijian Liu+3 more|arXiv (Cornell University)|Jun 18, 2020

Generative Adversarial Networks and Image Synthesis52 references100 citations

TL;DR

DiffAugment applies differentiable augmentations to both real and generated samples during GAN training to improve data efficiency and convergence, achieving strong results across architectures and datasets.

ABSTRACT

The performance of generative adversarial networks (GANs) heavily deteriorates given a limited amount of training data. This is mainly because the discriminator is memorizing the exact training set. To combat it, we propose Differentiable Augmentation (DiffAugment), a simple method that improves the data efficiency of GANs by imposing various types of differentiable augmentations on both real and fake samples. Previous attempts to directly augment the training data manipulate the distribution of real images, yielding little benefit; DiffAugment enables us to adopt the differentiable augmentation for the generated samples, effectively stabilizes training, and leads to better convergence. Experiments demonstrate consistent gains of our method over a variety of GAN architectures and loss functions for both unconditional and class-conditional generation. With DiffAugment, we achieve a state-of-the-art FID of 6.80 with an IS of 100.8 on ImageNet 128x128 and 2-4x reductions of FID given 1,000 images on FFHQ and LSUN. Furthermore, with only 20% training data, we can match the top performance on CIFAR-10 and CIFAR-100. Finally, our method can generate high-fidelity images using only 100 images without pre-training, while being on par with existing transfer learning algorithms. Code is available at https://github.com/mit-han-lab/data-efficient-gans.

Motivation & Objective

• Motivation to improve GAN data efficiency when training data are scarce.
• Prevent discriminator overfitting without distorting the target data distribution.
• Enable gradients to flow through augmentations to the generator to stabilize training.

Proposed method

• Apply the same differentiable augmentation T to both real and fake samples during D and G updates.
• Use simple augmentations: Translation, Cutout, and Color, and study their compositions.
• Ensure T is differentiable so gradients can back-propagate to G (Figure 4).
• Demonstrate that augmenting only real data or only discriminator inputs fails due to distribution shift or unbalanced training dynamics.
• Evaluate DiffAugment with BigGAN and StyleGAN2 across ImageNet, CIFAR, FFHQ, LSUN-Cat, and low-shot settings.

Experimental results

Research questions

• RQ1 Does applying differentiable augmentation to both real and generated samples stabilize GAN training under limited data?
• RQ2 Which augmentation types (and compositions) best improve data efficiency across architectures?
• RQ3 How does DiffAugment perform across unconditional and conditional GANs on diverse datasets and data regimes?

Key findings

• BigGAN with DiffAugment achieves IS 100.8 and FID 6.80 on ImageNet 128×128 without truncation.
• DiffAugment reduces FID by 2–4× on FFHQ and LSUN with 1k training images.
• With 20% of CIFAR-10/CIFAR-100 data, DiffAugment matches top performance; achieves strong low-shot results without pre-training (100 images).
• DiffAugment consistently improves StyleGAN2 and BigGAN baselines across 100%, 50%, and 25% data settings.
• Stronger augmentation policies reduce discriminator overfitting and improve convergence (Figure 6).
• DiffAugment remains effective with fixed augmentation policies comparable to concurrent adaptive augmentation methods (ADA).