[Paper Review] Data-Free Knowledge Distillation for Deep Neural Networks

Raphael Gontijo Lopes, Stefano Fenu+1 more|arXiv (Cornell University)|Oct 19, 2017

Generative Adversarial Networks and Image SynthesisComputer Science15 references214 citations

TL;DR

The paper presents a data-free knowledge distillation approach that reconstructs training-like data from model metadata to compress large neural networks without access to the original training data.

ABSTRACT

Recent advances in model compression have provided procedures for compressing large neural networks to a fraction of their original size while retaining most if not all of their accuracy. However, all of these approaches rely on access to the original training set, which might not always be possible if the network to be compressed was trained on a very large dataset, or on a dataset whose release poses privacy or safety concerns as may be the case for biometrics tasks. We present a method for data-free knowledge distillation, which is able to compress deep neural networks trained on large-scale datasets to a fraction of their size leveraging only some extra metadata to be provided with a pretrained model release. We also explore different kinds of metadata that can be used with our method, and discuss tradeoffs involved in using each of them.

Motivation & Objective

• Motivate compression when original training data cannot be released due to privacy or scale constraints.
• Propose a data-free distillation pipeline that uses activation-based metadata to reconstruct inputs.
• Explore multiple activation-record strategies to balance reconstruction quality and metadata requirements.
• Demonstrate scalability from MNIST to CelebA with varying model sizes.

Proposed method

• Train a teacher model and store activation records across layers as metadata.
• Regenerate input data by optimizing random noise to match stored activations using gradient-based inversion.
• Train a student network on reconstructed data using temperature-scaled activations as labels (knowledge distillation).
• Experiment with different activation-record strategies: top-layer statistics, all-layers statistics, and spectral (graph Fourier) methods.
• Optionally freeze dropout during reconstruction to preserve inter-layer dynamics.
• Assess performance across datasets and model families (MNIST fully-connected, MNIST LeNet-5, CelebA with AlexNet).

Experimental results

Research questions

• RQ1 Can a pretrained model be compressed without access to its original training data by using only activation metadata?
• RQ2 How do different activation-record strategies affect reconstruction quality and distillation accuracy?
• RQ3 What are the tradeoffs between simple statistical, all-layers, and spectral activation representations for data-free distillation?
• RQ4 How does data-free distillation scale to large datasets and architectures?
• RQ5 What metadata best preserves inter-layer dynamics during reconstruction?

Key findings

• Top-layer statistics alone yield modest accuracy (e.g., 68.75% on MNIST with fully connected teacher).
• All-layers statistics improve accuracy (e.g., 76.38% on MNIST FC; 85.61% on MNIST with LeNet-5 full reconstruction).
• All-layers spectral and layer-pairs spectral methods achieve the highest accuracies (e.g., 89.41% and 91.24% on MNIST FC; 90.28% and 92.47% on MNIST LeNet-5-half).
• CelebA experiments show data-free distillation can approach or exceed baseline in some setups (e.g., AlexNet-half with All-Layers Spectral 77.56%; Layer-Pairs Spectral 76.94%).
• The method scales to large datasets and models, illustrating practical viability for data-release-constrained compression.