[Paper Review] Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?

Zheng Li, Yuxuan Li+4 more|arXiv (Cornell University)|May 22, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper investigates knowledge distillation using synthetic images generated by state-of-the-art diffusion models in the absence of real training data. It proposes a simple, black-box-compatible method leveraging public diffusion models to generate proxy datasets, achieving state-of-the-art performance across multiple benchmarks—surprisingly, lower-fidelity images and weaker teacher models yield better results due to reduced domain shift and improved feature transfer.

ABSTRACT

Diffusion models have recently achieved astonishing performance in generating high-fidelity photo-realistic images. Given their huge success, it is still unclear whether synthetic images are applicable for knowledge distillation when real images are unavailable. In this paper, we extensively study whether and how synthetic images produced from state-of-the-art diffusion models can be used for knowledge distillation without access to real images, and obtain three key conclusions: (1) synthetic data from diffusion models can easily lead to state-of-the-art performance among existing synthesis-based distillation methods, (2) low-fidelity synthetic images are better teaching materials, and (3) relatively weak classifiers are better teachers. Code is available at https://github.com/zhengli97/DM-KD.

Motivation & Objective

• To investigate whether synthetic images from state-of-the-art diffusion models can effectively replace real data in knowledge distillation.
• To evaluate the performance of distillation using only synthetic data when real data is unavailable due to privacy or security constraints.
• To identify optimal configurations for synthetic data generation and teacher-student architecture selection in a data-free distillation setting.
• To understand the impact of image fidelity and teacher model capacity on distillation performance with synthetic data.

Proposed method

• Leveraging publicly available pre-trained conditional diffusion models (e.g., Stable Diffusion, DiT) to generate synthetic images as proxy training data for knowledge distillation.
• Using the pre-trained teacher model to produce soft labels for the synthetic images, which are then used to train the student model via standard distillation loss.
• Applying temperature scaling in the distillation process to smooth probability distributions and improve knowledge transfer.
• Employing class labels from the diffusion model as hard labels in joint training with soft labels to assess their impact.
• Systematically varying diffusion model hyperparameters such as guidance scale (s) and number of sampling steps (T) to evaluate their effect on distillation performance.
• Conducting ablation studies on temperature, label types (soft vs. hard), and teacher-student capacity gaps to isolate key factors.

Experimental results

Research questions

• RQ1 Can synthetic images generated by state-of-the-art diffusion models effectively replace real data for knowledge distillation in a data-free setting?
• RQ2 Does higher image fidelity in synthetic data lead to better distillation performance, or is lower fidelity more beneficial?
• RQ3 Is a stronger teacher model always better for distillation when using synthetic data, or can a weaker teacher yield superior results?
• RQ4 How do hyperparameters like guidance scale and sampling steps in diffusion models affect the quality and utility of synthetic data for distillation?
• RQ5 To what extent does the use of hard labels from the diffusion model’s class labels improve or degrade distillation performance compared to soft labels alone?

Key findings

• Synthetic data generated from diffusion models achieves state-of-the-art performance in data-free knowledge distillation across ImageNet-1K, ImageNet-100, CIFAR-100, and Flowers-102, even when there is no category overlap between synthetic and target datasets.
• Lower-fidelity synthetic images—produced with smaller guidance scale (s) and fewer sampling steps (T)—lead to better student model performance than higher-fidelity images, despite better ImageReward scores for the latter.
• Using a relatively weak teacher model (e.g., ResNet18) outperforms using a stronger one (e.g., ResNet50) when training on synthetic data, with a 3% accuracy gain observed on ImageNet when training ResNet34.
• Joint training with hard labels from the diffusion model’s class labels degrades performance compared to using only soft labels, indicating a domain shift between synthetic and real data.
• The optimal temperature hyperparameter for distillation is τ = 10, which yields the best performance across experiments.
• The performance of the student model improves with increasing numbers of synthetic images used for distillation, demonstrating scalability of the proposed method.