[Paper Review] Data-Free Adversarial Distillation

Gongfan Fang, Jie Song+4 more|arXiv (Cornell University)|Dec 23, 2019

Adversarial Robustness in Machine Learning39 references103 citations

TL;DR

The paper introduces Data-Free Adversarial Distillation (DFAD), a two-stage adversarial framework that crafts a data-free training signal for a student model by jointly using a generator and a teacher-student discriminator to minimize an optimizable upper bound on model discrepancy. It extends to semantic segmentation and achieves competitive results with data-driven methods.

ABSTRACT

Knowledge Distillation (KD) has made remarkable progress in the last few years and become a popular paradigm for model compression and knowledge transfer. However, almost all existing KD algorithms are data-driven, i.e., relying on a large amount of original training data or alternative data, which is usually unavailable in real-world scenarios. In this paper, we devote ourselves to this challenging problem and propose a novel adversarial distillation mechanism to craft a compact student model without any real-world data. We introduce a model discrepancy to quantificationally measure the difference between student and teacher models and construct an optimizable upper bound. In our work, the student and the teacher jointly act the role of the discriminator to reduce this discrepancy, when a generator adversarially produces some "hard samples" to enlarge it. Extensive experiments demonstrate that the proposed data-free method yields comparable performance to existing data-driven methods. More strikingly, our approach can be directly extended to semantic segmentation, which is more complicated than classification, and our approach achieves state-of-the-art results. Code and pretrained models are available at https://github.com/VainF/Data-Free-Adversarial-Distillation.

Motivation & Objective

• Motivate knowledge distillation when original training data is unavailable in real-world scenarios.
• Propose a data-free framework to approximate and minimize the teacher-student model discrepancy without real data.
• Develop an adversarial training mechanism that continually crafts hard samples to improve the student.
• Extend data-free distillation to semantic segmentation and demonstrate competitive performance.

Proposed method

• Define model discrepancy between teacher T and student S as D(T,S) and approximate it with a generator G that produces training samples.
• Use a two-stage adversarial process: imitation stage (minimize discrepancy using MAE loss on samples from G) and generation stage (maximize discrepancy by optimizing G with a stable log-MAE objective).
• Adopt MAE as the discrepancy loss to ensure stable gradients and prevent generator collapse in the absence of real data.
• Split generated samples into hard and easy types to bound the discrepancy and guide the generator to produce challenging, informative samples.
• Iteratively update S to imitate T on generated samples while updating G to produce harder samples, aiming for S to become functionally indistinguishable from T.
• Provide stability guidance (e.g., fixing k=5 imitation steps, using L_GEN-ADA for segmentation) to ensure robust training.

Experimental results

Research questions

• RQ1 Can a data-free distillation framework match the performance of data-driven KD methods across classification and segmentation tasks?
• RQ2 How can we quantify and minimize the discrepancy between teacher and student models without real data?
• RQ3 Can adversarially generated samples be used effectively to train a competitive student model in the absence of original data?
• RQ4 Does the proposed framework extend to segmentation tasks with competitive results?

Key findings

• The proposed DFAD framework yields competitive performance compared to data-driven distillation baselines on classification datasets.
• Among data-free methods, the approach achieves the highest accuracy on several classification benchmarks (classification results outperform other data-free approaches).
• The method extends naturally to semantic segmentation and achieves competitive mIoU scores on CamVid and NYUv2, surpassing other data-free methods.
• The generated samples remain diverse, addressing mode collapse and providing informative supervision during training.
• MAE-based discrepancy loss provides stable gradients and outperforms other loss choices (e.g., MSE, KLD) for the generator.