[Paper Review] Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion

Hongxu Yin, Pavlo Molchanov+6 more|arXiv (Cornell University)|Dec 18, 2019

Domain Adaptation and Few-Shot Learning75 references48 citations

TL;DR

DeepInversion synthesizes high-fidelity, class-conditional images from a pretrained CNN without any training data, enabling data-free knowledge transfer, pruning, and continual learning; Adaptive DeepInversion increases image diversity by exploiting teacher-student disagreements.

ABSTRACT

We introduce DeepInversion, a new method for synthesizing images from the image distribution used to train a deep neural network. We 'invert' a trained network (teacher) to synthesize class-conditional input images starting from random noise, without using any additional information about the training dataset. Keeping the teacher fixed, our method optimizes the input while regularizing the distribution of intermediate feature maps using information stored in the batch normalization layers of the teacher. Further, we improve the diversity of synthesized images using Adaptive DeepInversion, which maximizes the Jensen-Shannon divergence between the teacher and student network logits. The resulting synthesized images from networks trained on the CIFAR-10 and ImageNet datasets demonstrate high fidelity and degree of realism, and help enable a new breed of data-free applications - ones that do not require any real images or labeled data. We demonstrate the applicability of our proposed method to three tasks of immense practical importance -- (i) data-free network pruning, (ii) data-free knowledge transfer, and (iii) data-free continual learning. Code is available at https://github.com/NVlabs/DeepInversion

Motivation & Objective

• Motivate data-free knowledge transfer when training data is unavailable or private.
• Introduce DeepInversion to synthesize class-conditional images from a fixed teacher network.
• Enhance image diversity with Adaptive DeepInversion by leveraging teacher-student disagreement.
• Demonstrate data-free pruning, data-free knowledge transfer, and data-free continual learning on CIFAR-10 and ImageNet.

Proposed method

• Invert a trained CNN to synthesize class-conditional images starting from random noise.
• Regularize intermediate feature maps using per-layer batch normalization running statistics to match training data statistics.
• Introduce a feature distribution regularizer that aligns the synthesized images’ layer-wise means and variances with BN running statistics.
• In Adaptive DeepInversion, add a competition loss based on Jensen-Shannon divergence to encourage teacher-student disagreement and broaden image coverage.
• Apply synthesized images to data-free pruning, data-free knowledge transfer, and data-free continual learning tasks.
• Provide hardware-aware pruning by incorporating latency into filter importance to maintain performance with fewer resources.

Experimental results

Research questions

• RQ1 Can high-fidelity, class-conditional images be synthesized without access to the original training data?
• RQ2 To what extent can data-free synthesized images support knowledge transfer and pruning without real data?
• RQ3 Does introducing a student-in-the-loop via Adaptive DeepInversion increase image diversity and downstream task performance?
• RQ4 How does data-free knowledge transfer compare to dataset-dependent distillation on CIFAR-10 and ImageNet?
• RQ5 Can data-free methods enable continual learning without original data?

Key findings

• DeepInversion produces high-fidelity, class-conditional images using only BN running statistics from a pretrained network.
• Adaptive DeepInversion further improves image diversity by maximizing teacher–student disagreement.
• Data-free pruning achieves competitive accuracy and latency gains, reducing dependence on original data.
• Data-free knowledge transfer via synthesized data yields near-teacher performance on CIFAR-10 and substantial accuracy on ImageNet with minimal loss.
• Data-free continual learning improves incremental performance and approaches oracle methods that have access to original data.
• Overall, the approach enables practical data-free applications without requiring real images or labels.