[Paper Review] Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis

Zi Wang|arXiv (Cornell University)|Apr 10, 2021

Advanced Neural Network Applications37 references4 citations

TL;DR

This paper proposes a data-free knowledge distillation method that synthesizes high-quality pseudo training samples by modeling the intermediate feature space of a teacher network with a multivariate normal distribution, generating soft targets for noise optimization. The approach achieves state-of-the-art performance—99.08% accuracy on MNIST and 93.31% on CIFAR-10—without using any original training data, outperforming prior data-free KD methods by leveraging more generalized, label-mismatch-resistant soft targets from shallower layers.

ABSTRACT

Knowledge distillation (KD) has proved to be an effective approach for deep neural network compression, which learns a compact network (student) by transferring the knowledge from a pre-trained, over-parameterized network (teacher). In traditional KD, the transferred knowledge is usually obtained by feeding training samples to the teacher network to obtain the class probabilities. However, the original training dataset is not always available due to storage costs or privacy issues. In this study, we propose a novel data-free KD approach by modeling the intermediate feature space of the teacher with a multivariate normal distribution and leveraging the soft targeted labels generated by the distribution to synthesize pseudo samples as the transfer set. Several student networks trained with these synthesized transfer sets present competitive performance compared to the networks trained with the original training set and other data-free KD approaches.

Motivation & Objective

• To address the challenge of knowledge distillation when original training data is unavailable due to privacy or storage constraints.
• To improve the quality of synthesized transfer sets in data-free KD by modeling intermediate feature representations rather than directly modeling softmax outputs.
• To reduce label mismatch issues common in prior methods that use distributions like Dirichlet on softmax space.
• To enhance student network performance through more generalized soft targets derived from shallow, high-level feature spaces.
• To validate the effectiveness of the proposed method across multiple architectures and benchmark datasets.

Proposed method

• Model the output of an intermediate layer (e.g., FC-2) in the teacher network using a multivariate normal distribution to capture the feature distribution.
• Sample soft target labels from this multivariate normal distribution to guide the synthesis of pseudo training images.
• Optimize random noise inputs via backpropagation to minimize the distance between the teacher’s forward pass on the noise and the sampled soft targets.
• Use the optimized pseudo samples as a transfer set to train the student network via standard knowledge distillation with distillation loss.
• Incorporate an additional activation loss to encourage higher neuron activations, improving the quality of synthesized samples.
• Leverage t-SNE visualization to compare the clustering quality of soft targets generated by multivariate normal vs. Dirichlet distributions.

Experimental results

Research questions

• RQ1 Can modeling the intermediate feature space of a teacher network with a multivariate normal distribution improve the quality of synthesized pseudo samples in data-free KD?
• RQ2 Does using soft targets from shallower feature layers lead to better generalization and performance in student models compared to modeling the final softmax layer?
• RQ3 How does the label mismatch problem in prior methods—such as those using Dirichlet distributions on softmax outputs—affect performance, and can it be mitigated?
• RQ4 To what extent does adding an activation loss improve the optimization of pseudo samples and the resulting student model accuracy?
• RQ5 How does the proposed method compare to existing data-free KD approaches in terms of accuracy and robustness across different datasets and architectures?

Key findings

• The proposed method achieved 99.08% test accuracy on MNIST and 93.31% on CIFAR-10 using only the teacher network and no original training data, outperforming existing data-free KD methods.
• Modeling the feature space of the second-to-last fully connected layer (FC-2) instead of the softmax layer improved student accuracy by 0.12% on LeNet-5-HALF and 1.59% on AlexNet-HALF.
• The multivariate normal distribution generated more coherent and well-separated soft targets than the Dirichlet distribution, reducing label mismatch issues by avoiding per-class sampling that causes misalignment.
• The inclusion of the activation loss further improved performance, though the gain was smaller than that from modeling intermediate features, indicating that feature space modeling is the primary contributor to success.
• t-SNE visualization confirmed that soft targets from the multivariate normal distribution formed clearer, more separable clusters than those from the Dirichlet distribution, supporting better sample synthesis.
• The method demonstrated consistent performance gains across multiple architectures and datasets, validating its generalizability and robustness in data-free KD settings.