[Paper Review] Distilling portable Generative Adversarial Networks for Image Translation

Hanting Chen, Yunhe Wang+6 more|arXiv (Cornell University)|Mar 7, 2020

Generative Adversarial Networks and Image Synthesis29 references4 citations

TL;DR

This paper proposes a knowledge distillation framework to compress heavy Generative Adversarial Networks (GANs) for mobile deployment in image-to-image translation. By training a lightweight student generator and discriminator to mimic both the output and feature-level distributions of a large teacher GAN, the method achieves state-of-the-art performance with up to 93% fewer parameters and 93% lower FLOPs, while maintaining high image fidelity and perceptual quality.

ABSTRACT

Despite Generative Adversarial Networks (GANs) have been widely used in various image-to-image translation tasks, they can be hardly applied on mobile devices due to their heavy computation and storage cost. Traditional network compression methods focus on visually recognition tasks, but never deal with generation tasks. Inspired by knowledge distillation, a student generator of fewer parameters is trained by inheriting the low-level and high-level information from the original heavy teacher generator. To promote the capability of student generator, we include a student discriminator to measure the distances between real images, and images generated by student and teacher generators. An adversarial learning process is therefore established to optimize student generator and student discriminator. Qualitative and quantitative analysis by conducting experiments on benchmark datasets demonstrate that the proposed method can learn portable generative models with strong performance.

Motivation & Objective

• To address the challenge of deploying heavy GANs on mobile devices due to high computational and memory costs.
• To extend knowledge distillation—previously used for classification—to generative models, particularly GANs for image-to-image translation.
• To jointly compress both the generator and discriminator by transferring knowledge from a teacher GAN to a student GAN.
• To improve the performance of lightweight student models by incorporating perceptual, L1, and triplet losses, along with adversarial training.
• To demonstrate generalization across paired (pix2pix) and unpaired (CycleGAN) image translation tasks.

Proposed method

• A student generator is trained using L1 and perceptual losses to match the output images and feature representations of the teacher generator.
• A student discriminator is introduced to distinguish between real images and generated images from both the teacher and student generators.
• The student GAN is optimized via a minimax adversarial objective, enabling the student to learn the data distribution from the teacher.
• The total loss for the student generator includes L1, perceptual, and knowledge distillation losses from the teacher.
• The total loss for the student discriminator includes adversarial losses based on real images and generated images from both teacher and student networks.
• The framework is applied to both paired (pix2pix) and unpaired (CycleGAN) image translation tasks with consistent performance gains.

Experimental results

Research questions

• RQ1 Can knowledge distillation be effectively adapted to compress GANs for image-to-image translation, rather than just classification models?
• RQ2 Does jointly training a student discriminator improve the generative quality of a compressed student generator?
• RQ3 Can a student GAN with significantly fewer parameters and FLOPs match the performance of a heavy teacher GAN in image translation tasks?
• RQ4 How do different loss components—L1, perceptual, and triplet—contribute to the performance of the distilled student model?
• RQ5 Does the proposed method generalize across different GAN architectures, such as paired and unpaired image translation frameworks?

Key findings

• The proposed method achieved a student generator with only 715.65K parameters and 3.19G FLOPs, reducing the teacher’s 11.38M parameters and 47.19G FLOPs by 93.7% and 93.2%, respectively.
• On the horse→zebra and summer→winter translation tasks, the distilled student GAN achieved image quality comparable to the teacher GAN, outperforming models trained from scratch and vanilla distillation.
• The ablation study showed that combining L1, perceptual, and triplet losses with adversarial training yielded the highest FID and LPIPS scores, proving the effectiveness of each component.
• In the unpaired CycleGAN setting, the student model with 1/16 the parameters of the teacher achieved similar performance, demonstrating strong generalization.
• The student discriminator significantly improved the realism of generated images, reducing blur and improving fidelity compared to models without a student discriminator.
• The method achieved a 52.22 FID score on the Cityscapes dataset, outperforming the baseline student (53.15) and vanilla distillation (52.50), while using far fewer parameters.