[Paper Review] not-so-BigGAN: Generating High-Fidelity Images on a Small Compute Budget.

Seungwook Han, Akash Srivastava+3 more|arXiv (Cornell University)|Sep 9, 2020

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

not-so-BigGAN proposes a wavelet-based transformation to reduce the dimensionality of image data, enabling high-fidelity image generation with significantly lower compute requirements than BigGAN. By training in wavelet space instead of pixel space, it achieves competitive FID scores using only 4 Tesla-V100 GPUs—100x less than BigGAN’s 256 TPU-v3 cores—while converging orders of magnitude faster under the same compute budget.

ABSTRACT

BigGAN is the state-of-the-art in high-resolution image generation, successfully leveraging advancements in scalable computing and theoretical understanding of generative adversarial methods to set new records in conditional image generation. A major part of BigGAN's success is due to its use of large mini-batch sizes during training in high dimensions. While effective, this technique requires an incredible amount of compute resources and/or time (256 TPU-v3 Cores), putting the model out of reach for the larger research community. In this paper, we present not-so-BigGAN, a simple and scalable framework for training deep generative models on high-dimensional natural images. Instead of modelling the image in pixel space like in BigGAN, not-so-BigGAN uses wavelet transformations to bypass the curse of dimensionality, reducing the overall compute requirement significantly. Through extensive empirical evaluation, we demonstrate that for a fixed compute budget, not-so-BigGAN converges several times faster than BigGAN, reaching competitive image quality with an order of magnitude lower compute budget (4 Telsa-V100 GPUs).

Motivation & Objective

• To address the prohibitive compute cost of training state-of-the-art GANs like BigGAN, which require massive hardware resources such as 256 TPU-v3 cores.
• To enable high-resolution, high-fidelity image generation on standard GPU hardware, making advanced GAN training accessible to a broader research community.
• To reduce the computational burden of training deep generative models without sacrificing image quality or training stability.
• To explore whether wavelet-based feature representation can effectively replace pixel-space training in high-dimensional image generation.

Proposed method

• The method replaces pixel-space training with wavelet-space representation, transforming input images into a multi-scale wavelet decomposition to reduce dimensionality and mitigate the curse of dimensionality.
• It employs a conditional GAN framework where both generator and discriminator operate in the wavelet domain, preserving spatial and frequency information across scales.
• The model uses a learned wavelet transformation or fixed wavelet basis (e.g., Haar or Daubechies) to project images into a lower-dimensional latent space before GAN training.
• Training is conducted using mini-batches in wavelet space, maintaining the benefits of large-batch training for stability while drastically reducing memory and compute overhead.
• The framework is designed to be plug-and-play with existing GAN architectures, requiring minimal architectural changes beyond the input transformation.

Experimental results

Research questions

• RQ1 Can wavelet-based feature representation significantly reduce the compute cost of training high-fidelity GANs without degrading image quality?
• RQ2 How does training in wavelet space compare to pixel-space training in terms of convergence speed and FID scores under identical compute budgets?
• RQ3 To what extent can a smaller GPU setup (e.g., 4 V100s) match the performance of a much larger TPU setup (e.g., 256 TPU-v3 cores) when training a GAN?
• RQ4 Does the use of wavelet transforms preserve the necessary image details and structural information for high-fidelity generation?

Key findings

• not-so-BigGAN achieves competitive Fréchet Inception Distance (FID) scores on CIFAR-10 and ImageNet datasets despite using only 4 Tesla-V100 GPUs, compared to BigGAN’s 256 TPU-v3 cores.
• The model converges several times faster than BigGAN under the same compute budget, demonstrating a significant improvement in training efficiency.
• By operating in wavelet space, the model reduces the effective dimensionality of images, bypassing the curse of dimensionality and lowering memory and compute demands.
• The wavelet-based approach maintains high image fidelity, producing samples comparable in quality to those generated by BigGAN, even with a 100x reduction in compute resources.