[Paper Review] Small-GAN: Speeding Up GAN Training Using Core-sets

Samarth Sinha, Han Zhang+4 more|arXiv (Cornell University)|Oct 29, 2019

Generative Adversarial Networks and Image SynthesisComputer Science69 references33 citations

TL;DR

Proposes Core-set based sampling to emulate large-batch GAN training, yielding faster training with similar or improved performance across several GAN variants and datasets.

ABSTRACT

Recent work by Brock et al. (2018) suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. Unfortunately, using large batches is slow and expensive on conventional hardware. Thus, it would be nice if we could generate batches that were effectively large though actually small. In this work, we propose a method to do this, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of 'real' images, we create a cached dataset of Inception activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected activations at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it allows GANs to reach a new state of the art in anomaly detection.

Motivation & Objective

• Motivate the use of large mini-batches in GAN training and address the computational bottlenecks they entail.
• Introduce Core-set sampling to create smaller batches that cover the same modes as larger batches.
• Make Core-set sampling practical for GANs by operating on both prior samples and high-level embeddings.
• Demonstrate the method's effectiveness across multiple GAN variants and datasets (CIFAR, LSUN, ImageNet) and in anomaly detection.
• Show that core-set augmentation can reduce training time and memory usage while improving or matching performance metrics.

Proposed method

• Sample a large batch n from the prior and perform Core-set selection to obtain a smaller batch k.
• Preprocess target data by computing Inception embeddings and apply a random projection to lower dimensions before Core-set selection.
• Use a greedy k-center algorithm as an efficient approximate Core-set sampler to select representative points.
• Apply Core-set sampling to both the prior and the target (embedding) distributions to create effectively large batches.
• Train GANs with standard objectives, but with Core-set-derived batches instead of random mini-batches.
• Maintain a cached Embedding dataset to speed up runtime Core-set decisions.

Experimental results

Research questions

• RQ1 Can Core-set sampling replicate the coverage of large batches in GAN training with reduced computation?
• RQ2 Does Small-GAN improve or preserve GAN performance (FID, mode coverage, anomaly detection metrics) with smaller actual batch sizes?
• RQ3 Is the method robust across GAN variants (SN-GAN, SAGAN) and datasets (CIFAR, LSUN, ImageNet)?

Key findings

• Core-set sampling improves FID scores across CIFAR and LSUN for multiple GAN variants at fixed batch sizes.
• The method reduces mode dropping on Gaussian mixtures, indicating better mode coverage.
• Core-set augmented GANs improve anomaly detection results in MEG-based experiments.
• Timing analysis shows only a small overhead (~0.024 seconds per gradient step) for performing Core-set sampling.
• On ImageNet with SAGAN, Core-set sampling improves FID from 19.40 to 17.33.
• Across datasets, larger actual batches yield similar improvements, but Core-set sampling provides gains without proportional resource increases.