[Paper Review] Dual Student Networks for Data-Free Model Stealing

James Beetham, Navid Kardan+2 more|arXiv (Cornell University)|Sep 18, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

This paper proposes Dual Student Networks, a data-free model stealing method that trains two student models to mimic a target model, using their disagreement to guide a generator in producing diverse, hard-to-classify samples. By leveraging the gradients of both students as a proxy for the target model’s gradients, the method achieves more accurate gradient estimation and higher student model accuracy than prior data-free approaches, while also serving as a superior proxy for transfer-based adversarial attacks.

ABSTRACT

Existing data-free model stealing methods use a generator to produce samples in order to train a student model to match the target model outputs. To this end, the two main challenges are estimating gradients of the target model without access to its parameters, and generating a diverse set of training samples that thoroughly explores the input space. We propose a Dual Student method where two students are symmetrically trained in order to provide the generator a criterion to generate samples that the two students disagree on. On one hand, disagreement on a sample implies at least one student has classified the sample incorrectly when compared to the target model. This incentive towards disagreement implicitly encourages the generator to explore more diverse regions of the input space. On the other hand, our method utilizes gradients of student models to indirectly estimate gradients of the target model. We show that this novel training objective for the generator network is equivalent to optimizing a lower bound on the generator's loss if we had access to the target model gradients. We show that our new optimization framework provides more accurate gradient estimation of the target model and better accuracies on benchmark classification datasets. Additionally, our approach balances improved query efficiency with training computation cost. Finally, we demonstrate that our method serves as a better proxy model for transfer-based adversarial attacks than existing data-free model stealing methods.

Motivation & Objective

• To address the challenge of data-free model stealing, where no access to training data or model weights is available.
• To improve gradient estimation of the target model without direct access to its parameters or gradients.
• To enhance the diversity and representativeness of generated training samples for student model distillation.
• To develop a more effective proxy model for transfer-based adversarial attacks than existing data-free methods.
• To reduce reliance on synthetic datasets or pre-trained models while maintaining high performance.

Proposed method

• Two student models are trained symmetrically to match the target model’s outputs on generated samples.
• A generator is trained to produce samples on which the two student models disagree, implicitly targeting hard, boundary-like samples.
• Disagreement between students indicates at least one student is incorrect relative to the target model, guiding the generator toward challenging input regions.
• Gradients from both student models are used to estimate the target model’s gradients, avoiding reliance on explicit gradient estimation techniques.
• The generator’s objective is shown to be equivalent to optimizing a lower bound on the true target model gradient loss, enabling direct optimization without queries.
• The method replaces the standard student-target min-max objective with a dual-student proxy, improving alignment between student and target model gradients.

Experimental results

Research questions

• RQ1 Can training two student models in parallel improve the quality and diversity of generated samples in data-free model stealing?
• RQ2 Does using gradients from two student models provide a better approximation of the target model’s gradients than existing data-free methods?
• RQ3 Can the disagreement between two students serve as an effective signal to guide the generator toward hard, boundary samples?
• RQ4 How does the proposed method compare in accuracy and query efficiency to state-of-the-art data-free model stealing baselines?
• RQ5 To what extent does the student model trained via this method serve as a better proxy for transfer-based adversarial attacks?

Key findings

• The Dual Student method achieves higher student model accuracy than DFME and DFMS-HL on CIFAR-10, with a 62.35% untargeted FGSM attack success rate compared to 56.84% for DFME.
• On PGD attacks, the method achieves 91.04% untargeted success rate, outperforming the white-box attack on the same target model (96.78%) in the untargeted setting.
• The method reduces the need for synthetic datasets or pretraining, enabling training from scratch with no external data or models.
• The generator’s objective is theoretically equivalent to optimizing a lower bound on the target model gradient loss, enabling direct optimization without queries.
• The approach improves query efficiency while balancing training computation cost, making it practical for real-world deployment.
• The student model produced by the method serves as a superior proxy for transfer-based adversarial attacks, especially under FGSM, where it outperforms both white-box and proxy models trained on real data.