[Paper Review] Zero-shot Knowledge Transfer via Adversarial Belief Matching

Paul Micaelli, Amos Storkey|arXiv (Cornell University)|May 23, 2019

Adversarial Robustness in Machine LearningComputer Science43 references33 citations

TL;DR

The paper trains a student to imitate a teacher without any data by using an adversarial generator to create pseudo inputs that maximize divergence between teacher and student, enabling zero-shot distillation especially on CIFAR-10 and SVHN.

ABSTRACT

Performing knowledge transfer from a large teacher network to a smaller student is a popular task in modern deep learning applications. However, due to growing dataset sizes and stricter privacy regulations, it is increasingly common not to have access to the data that was used to train the teacher. We propose a novel method which trains a student to match the predictions of its teacher without using any data or metadata. We achieve this by training an adversarial generator to search for images on which the student poorly matches the teacher, and then using them to train the student. Our resulting student closely approximates its teacher for simple datasets like SVHN, and on CIFAR10 we improve on the state-of-the-art for few-shot distillation (with 100 images per class), despite using no data. Finally, we also propose a metric to quantify the degree of belief matching between teacher and student in the vicinity of decision boundaries, and observe a significantly higher match between our zero-shot student and the teacher, than between a student distilled with real data and the teacher. Code available at: https://github.com/polo5/ZeroShotKnowledgeTransfer

Motivation & Objective

• Motivate and enable knowledge transfer from a large teacher to a smaller student without access to the training data.
• Develop an adversarial data generation process that identifies inputs where teacher and student disagree most.
• Demonstrate zero-shot distillation on standard vision benchmarks (SVHN, CIFAR-10) and compare to data-dependent baselines.
• Introduce a belief-matching metric near decision boundaries to quantify teacher-student alignment.

Proposed method

• Define a pretrained teacher T(x) and a student S(x;θ) with a generator G(z;φ) producing pseudo inputs x_p from noise z~N(0,I).
• Use forward KL divergence D_KL(T(x_p) || S(x_p)) as the main loss to train the student on pseudo data.
• Update G to maximize D_KL(T(x_p) || S(x_p)) so it finds inputs where teacher and student diverge.
• Iteratively train G (n_G steps) and then train S (n_S steps) on the fixed x_p to align with the teacher.
• Optionally include an attention-based regularization term across activation blocks to encourage feature alignment (β term).
• Rely on high-entropy student outputs to resist easy fooling by the generator and promote diverse, informative pseudo samples.

Experimental results

Research questions

• RQ1 Can a neural student learn to match a teacher without any access to real training data or metadata?
• RQ2 Does adversarially generated pseudo data enable effective zero-shot distillation on standard datasets?
• RQ3 How close is the zero-shot student to the teacher near decision boundaries, and how can this be quantified?
• RQ4 What is the impact of auxiliary losses (e.g., attention-based alignment) on zero-shot knowledge transfer?

Key findings

• The zero-shot method allows the student to closely approximate the teacher without using real data, achieving competitive results on CIFAR-10 and SVHN.
• On CIFAR-10, zero-shot distillation with the proposed loss achieves 83.69% test accuracy, improving over prior few-shot methods, and reaches 85.91% after fine-tuning with 100 images per class.
• Using the proposed loss with attention terms yields better performance than using the base KL loss alone, across several teacher–student pairs.
• The method remains robust to hyperparameter variations and dataset shifts, requiring no validation data for hyperparameter tuning in zero-shot settings.
• Compared to zero-shot competitors, the approach demonstrates a notable accuracy advantage on CIFAR-10 with similar or smaller model counts.
• A metric based on transition curves near decision boundaries shows that the zero-shot student more closely mirrors the teacher’s beliefs than a student distilled with real data.