[Paper Review] When, Why, and Which Pretrained GANs Are Useful?

Timofey Grigoryev, Andrey Voynov+1 more|arXiv (Cornell University)|Feb 17, 2022

Advanced Neural Network Applications5 citations

TL;DR

This paper investigates when, why, and which pretrained GANs are effective for finetuning, showing that pretrained generators enhance mode coverage while discriminators provide a guiding gradient field, leading to better diversity. It demonstrates that ImageNet-pretrained GANs—despite low sample fidelity—are optimal starting points for finetuning, and proposes a recall-based metric to automatically select the best checkpoint.

ABSTRACT

The literature has proposed several methods to finetune pretrained GANs on new datasets, which typically results in higher performance compared to training from scratch, especially in the limited-data regime. However, despite the apparent empirical benefits of GAN pretraining, its inner mechanisms were not analyzed in-depth, and understanding of its role is not entirely clear. Moreover, the essential practical details, e.g., selecting a proper pretrained GAN checkpoint, currently do not have rigorous grounding and are typically determined by trial and error. This work aims to dissect the process of GAN finetuning. First, we show that initializing the GAN training process by a pretrained checkpoint primarily affects the model's coverage rather than the fidelity of individual samples. Second, we explicitly describe how pretrained generators and discriminators contribute to the finetuning process and explain the previous evidence on the importance of pretraining both of them. Finally, as an immediate practical benefit of our analysis, we describe a simple recipe to choose an appropriate GAN checkpoint that is the most suitable for finetuning to a particular target task. Importantly, for most of the target tasks, Imagenet-pretrained GAN, despite having poor visual quality, appears to be an excellent starting point for finetuning, resembling the typical pretraining scenario of discriminative computer vision models.

Motivation & Objective

• Understand the mechanisms behind the success of GAN finetuning with pretrained models, especially in low-data regimes.
• Clarify the distinct roles of pretrained generators and discriminators in improving model coverage and generalization.
• Address the lack of rigorous criteria for selecting pretrained GAN checkpoints, which are currently chosen via trial and error.
• Provide a practical, automatic method to select the most suitable pretrained GAN checkpoint for a given target dataset.
• Demonstrate that ImageNet-pretrained GANs, despite poor visual quality, are highly effective for finetuning, mirroring discriminative model pretraining practices.

Proposed method

• Analyze the impact of pretrained GAN checkpoints on coverage and fidelity using synthetic and real data experiments.
• Decompose the contributions of pretrained generators (mode coverage) and discriminators (gradient field guidance) during finetuning.
• Propose a proxy metric based on FID, KID, Precision, and Recall to evaluate source dataset similarity for transfer learning.
• Use generated samples from pretrained models as a practical alternative to real source data for metric computation.
• Evaluate the predictive power of each metric by comparing their ability to identify the best checkpoint (lowest FID) across multiple target datasets.
• Validate findings on StyleGAN2 with diverse pretrained checkpoints, including ImageNet-based models.

Experimental results

Research questions

• RQ1 How does initializing GAN training with a pretrained checkpoint affect model coverage versus sample fidelity?
• RQ2 What specific roles do pretrained generators and discriminators play in improving finetuning performance?
• RQ3 Why is it beneficial to initialize both generator and discriminator during GAN finetuning?
• RQ4 Can recall or other distributional metrics reliably predict the best pretrained GAN checkpoint for a target dataset?
• RQ5 Is ImageNet-pretrained GAN a strong starting point for finetuning, despite its low sample quality?

Key findings

• Pretrained GAN checkpoints primarily improve model coverage rather than the fidelity of individual generated samples.
• A pretrained generator ensures initial diversity by covering multiple modes of the target distribution, while a pretrained discriminator provides a gradient field that guides samples toward correct modes.
• Initializing both generator and discriminator leads to better mode coverage and reduced risk of mode collapse, explaining the empirical success of full-model pretraining.
• For most target tasks, ImageNet-pretrained GANs are the optimal starting point for finetuning, even though their generated samples have low visual quality.
• The recall metric is a reliable predictor of finetuning success and can guide the automatic selection of the best pretrained checkpoint, outperforming FID and Precision in this role.
• Experiments on StyleGAN2 show that ImageNet-based pretraining leads to lower FID scores (e.g., 2.56 vs. 2.97 on L. Bedroom) and better image inversion performance (lower LPIPS and F-error), confirming its practical superiority.