[Paper Review] Transferring GANs: generating images from limited data

Yaxing Wang, Chenshen Wu+4 more|arXiv (Cornell University)|May 4, 2018

Generative Adversarial Networks and Image Synthesis47 references21 citations

TL;DR

This paper investigates transfer learning for generative adversarial networks (GANs), demonstrating that initializing GANs with pre-trained models from dense, limited-domain datasets (e.g., LSUN) significantly accelerates convergence and improves image quality—especially under data scarcity. Key results show that transferring the discriminator is more critical than the generator, and dense datasets outperform diverse ones like ImageNet for generation tasks.

ABSTRACT

Transferring the knowledge of pretrained networks to new domains by means of finetuning is a widely used practice for applications based on discriminative models. To the best of our knowledge this practice has not been studied within the context of generative deep networks. Therefore, we study domain adaptation applied to image generation with generative adversarial networks. We evaluate several aspects of domain adaptation, including the impact of target domain size, the relative distance between source and target domain, and the initialization of conditional GANs. Our results show that using knowledge from pretrained networks can shorten the convergence time and can significantly improve the quality of the generated images, especially when the target data is limited. We show that these conclusions can also be drawn for conditional GANs even when the pretrained model was trained without conditioning. Our results also suggest that density may be more important than diversity and a dataset with one or few densely sampled classes may be a better source model than more diverse datasets such as ImageNet or Places.

Motivation & Objective

• To investigate whether transfer learning—commonly used in discriminative models—can be effectively applied to generative models, particularly GANs.
• To evaluate the impact of source domain characteristics (e.g., diversity, density, class distribution) on transfer performance in GANs.
• To determine whether pre-trained unconditional GANs can be successfully adapted to conditional GANs for improved image generation.
• To analyze the relative importance of transferring the generator versus the discriminator in conditional and unconditional GANs.
• To assess the effectiveness of transfer learning in low-data regimes, where few labeled images are available for training.

Proposed method

• Fine-tuning pre-trained GANs (e.g., trained on ImageNet, Places, LSUN, CelebA) by initializing the generator and discriminator with pre-trained weights.
• Evaluating transfer performance using FID (Fréchet Inception Distance) and human evaluation for image realism on LSUN 10-class subsets.
• Training conditional GANs (AC-GAN) using pre-trained models from unconditional GANs, even when the source model lacked conditioning.
• Comparing training from scratch versus transfer learning across varying target dataset sizes (100, 1k, 10k images per class).
• Using a psychophysical experiment where human subjects judged which of two generated images (from scratch vs. pre-trained) appeared more realistic.
• Analyzing the influence of source-target domain distance and class density on transferability, particularly in low-data regimes.

Experimental results

Research questions

• RQ1 Can transfer learning from pre-trained GANs improve image generation quality and convergence speed when training data is limited?
• RQ2 How does the choice of source dataset (e.g., ImageNet, Places, LSUN, CelebA) affect transfer performance in GANs?
• RQ3 Is transferring the discriminator more beneficial than transferring the generator in GANs, and does transferring both yield the best results?
• RQ4 Can pre-trained unconditional GANs be effectively adapted to conditional GANs, even when the source model was not trained with conditioning?
• RQ5 Does dataset density (images per class) matter more than diversity for transferable features in GANs?

Key findings

• Pre-trained GANs significantly reduce training time and improve image quality, especially when target data is limited, as shown by lower FID scores (e.g., 119.1 vs. 162.9 at 100 images per class on LSUN).
• Transferring the discriminator alone yields better performance than transferring the generator alone, and transferring both networks produces the best results.
• Despite ImageNet’s high diversity, it performs poorly as a source for GAN transfer due to low class density; LSUN, with higher density, outperforms it significantly.
• Human evaluation shows that pre-trained GANs generate more realistic images—preferred in 67% of cases at 1k images per class—confirming FID trends.
• AC-GANs fine-tuned from pre-trained models achieve higher classifier accuracy (e.g., 65.9% vs. 52.7% at 1k images per class) and lower FID, indicating better distributional alignment.
• The study reveals that for GANs, high class density is more critical than diversity for transferability, challenging the assumption that diverse datasets like ImageNet are optimal.