[Paper Review] On Leveraging Pretrained GANs for Generation with Limited Data

Miaoyun Zhao, Yulai Cong+1 more|arXiv (Cornell University)|Feb 26, 2020

Generative Adversarial Networks and Image Synthesis63 references50 citations

TL;DR

This paper transfers low-level filters from pretrained GANs on large datasets to data-scarce domains, freezes them, and adds a small target-specific network with adaptive filter modulation (AdaFM) to improve generation with limited data, achieving better FID scores than baselines.

ABSTRACT

Recent work has shown generative adversarial networks (GANs) can generate highly realistic images, that are often indistinguishable (by humans) from real images. Most images so generated are not contained in the training dataset, suggesting potential for augmenting training sets with GAN-generated data. While this scenario is of particular relevance when there are limited data available, there is still the issue of training the GAN itself based on that limited data. To facilitate this, we leverage existing GAN models pretrained on large-scale datasets (like ImageNet) to introduce additional knowledge (which may not exist within the limited data), following the concept of transfer learning. Demonstrated by natural-image generation, we reveal that low-level filters (those close to observations) of both the generator and discriminator of pretrained GANs can be transferred to facilitate generation in a perceptually-distinct target domain with limited training data. To further adapt the transferred filters to the target domain, we propose adaptive filter modulation (AdaFM). An extensive set of experiments is presented to demonstrate the effectiveness of the proposed techniques on generation with limited data.

Motivation & Objective

• Motivate transferring information from pretrained GANs to facilitate generation in target domains with limited data.
• Identify which low-level filters are generally transferable in GANs across domains.
• Propose a compact domain-specific head and AdaFM to adapt transferred filters to the target domain.
• Demonstrate improved generation performance on multiple small datasets using transferred filters and AdaFM.

Proposed method

• Reuse low-level generator and discriminator filters from a pretrained GAN (GP-GAN on ImageNet) as the general part.
• Replace the high-level domain-specific part with a small tailored network (SmallHead) enabling style mixing.
• Introduce adaptive filter modulation (AdaFM) to modulate transferred filters via learnable gamma and beta parameters.
• Train the domain-specific head and AdaFM components on limited target data while keeping the general part frozen.
• Evaluate using Fréchet Inception Distance (FID) across multiple target datasets and compare with TransferGAN and Scratch baselines.

Experimental results

Research questions

• RQ1 Can low-level filters from pretrained GANs generalize to perceptually distinct target domains with limited data?
• RQ2 Does freezing transferred low-level filters plus a compact domain-specific head improve stability and quality of generation under data scarcity?
• RQ3 Does AdaFM provide measurable gains by adapting transferred filters to the target domain?
• RQ4 How does the proposed approach compare against existing transfer methods (e.g., TransferGAN) and training from scratch on limited data?

Key findings

• Transferring low-level generator/discriminator filters from a pretrained GAN improves generation quality and training efficiency on target domains with limited data.
• A compact tailored high-level network (SmallHead) helps reduce overfitting and enables style mixing.
• AdaFM further boosts performance by adaptively modulating transferred filters to the target domain.
• On CelebA, Flowers, Cars, and Cathedral, the proposed method achieves lower FID scores than TransferGAN and Scratch after 60,000 iterations.
• The method remains stable on extremely limited data (1K and 25 samples) and shows notable gains from AdaFM and the designed architecture.