[Paper Review] Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs

Sangwoo Mo, Minsu Cho+1 more|arXiv (Cornell University)|Feb 25, 2020

Generative Adversarial Networks and Image SynthesisComputer Science47 references122 citations

TL;DR

Freeze the lower layers of the discriminator while fine-tuning the upper layers (FreezeD) to transfer GANs to new domains; this simple baseline consistently improves FID over prior transfer-learning methods across unconditional and conditional GANs.

ABSTRACT

Generative adversarial networks (GANs) have shown outstanding performance on a wide range of problems in computer vision, graphics, and machine learning, but often require numerous training data and heavy computational resources. To tackle this issue, several methods introduce a transfer learning technique in GAN training. They, however, are either prone to overfitting or limited to learning small distribution shifts. In this paper, we show that simple fine-tuning of GANs with frozen lower layers of the discriminator performs surprisingly well. This simple baseline, FreezeD, significantly outperforms previous techniques used in both unconditional and conditional GANs. We demonstrate the consistent effect using StyleGAN and SNGAN-projection architectures on several datasets of Animal Face, Anime Face, Oxford Flower, CUB-200-2011, and Caltech-256 datasets. The code and results are available at https://github.com/sangwoomo/FreezeD.

Motivation & Objective

• Motivate a simple, robust transfer-learning baseline for GANs under limited data and distribution shift.
• Show that freezing discriminator lower layers yields strong performance across unconditional and conditional GANs.
• Demonstrate consistency and stability of FreezeD on diverse datasets and architectures.
• Compare FreezeD to prior methods (fine-tuning, GLO, MineGAN, L2-SP, feature distillation) and highlight practical advantages.

Proposed method

• Split the discriminator into a feature extractor (lower layers) and a classifier (upper layers) and only fine-tune the classifier.
• Apply this FreezeD baseline to StyleGAN pre-trained on FFHQ and SNGAN-projection pre-trained on ImageNet.
• Evaluate using FID on multiple target datasets (Animal Face, Anime Face, Oxford Flower, CUB-200-2011, Caltech-256).
• Compare FreezeD with prior transfer methods including Fine-tuning, Scale/Shift, GLO, MineGAN, L2-SP, and Feature Distillation.
• Provide ablations on which discriminator layers to freeze and discuss stability and diversity implications.

Experimental results

Research questions

• RQ1 Does freezing the lower layers of the discriminator during fine-tuning improve transfer learning for GANs over naive fine-tuning?
• RQ2 How does FreezeD perform across unconditional and conditional GANs on diverse target datasets with limited data?
• RQ3 How does FreezeD compare to existing transfer-learning methods in terms of FID and training stability?

Key findings

• FreezeD consistently improves FID over vanilla fine-tuning across unconditional and conditional GANs.
• Freezing the discriminator lower layers (up to the appropriate depth) stabilizes training and yields better best/final FID scores on several datasets.
• For StyleGAN transferred to Animal Face and Anime Face, FreezeD outperforms prior methods such as Scale/Shift, GLO, MineGAN, and L2-SP in most settings.
• For SNGAN-projection transferred to Oxford Flower, CUB-200-2011, and Caltech-256, FreezeD improves best/final FID in several cases, though stability can vary by dataset.
• Feature distillation shows comparable results to FreezeD but is generally slower, while FreezeD remains a simpler, effective baseline.
• Qualitative results indicate FreezeD generates more class-consistent samples and preserves latent-code semantics.