[Paper Review] Private Model Compression via Knowledge Distillation

Ji Wang, Weidong Bao+4 more|arXiv (Cornell University)|Nov 13, 2018

Privacy-Preserving Technologies in Data22 references20 citations

TL;DR

This paper proposes Rona, a private model compression framework that uses differentially private knowledge distillation to train compact, efficient student models from large teacher models without exposing sensitive user data. By adaptively bounding and perturbing distilled knowledge and employing query-efficient sample selection, Rona achieves up to 20× model compression and 19× speed-up on SVHN with only 0.97% accuracy loss while guaranteeing strong privacy (e.g., (9.83, 10⁻⁶)-differential privacy).

ABSTRACT

The soaring demand for intelligent mobile applications calls for deploying powerful deep neural networks (DNNs) on mobile devices. However, the outstanding performance of DNNs notoriously relies on increasingly complex models, which in turn is associated with an increase in computational expense far surpassing mobile devices' capacity. What is worse, app service providers need to collect and utilize a large volume of users' data, which contain sensitive information, to build the sophisticated DNN models. Directly deploying these models on public mobile devices presents prohibitive privacy risk. To benefit from the on-device deep learning without the capacity and privacy concerns, we design a private model compression framework RONA. Following the knowledge distillation paradigm, we jointly use hint learning, distillation learning, and self learning to train a compact and fast neural network. The knowledge distilled from the cumbersome model is adaptively bounded and carefully perturbed to enforce differential privacy. We further propose an elegant query sample selection method to reduce the number of queries and control the privacy loss. A series of empirical evaluations as well as the implementation on an Android mobile device show that RONA can not only compress cumbersome models efficiently but also provide a strong privacy guarantee. For example, on SVHN, when a meaningful $(9.83,10^{-6})$-differential privacy is guaranteed, the compact model trained by RONA can obtain 20$ imes$ compression ratio and 19$ imes$ speed-up with merely 0.97% accuracy loss.

Motivation & Objective

• Address the dual challenge of deploying large, accurate deep neural networks (DNNs) on resource-constrained mobile devices while preserving user data privacy.
• Overcome the limitations of existing knowledge distillation methods, which do not provide formal privacy guarantees despite using sensitive training data.
• Develop a framework that enables on-device inference using compact student models trained on public data, without exposing the teacher model or sensitive user data.
• Ensure provable privacy through differential privacy by carefully perturbing distilled knowledge while maintaining high model accuracy and efficiency.

Proposed method

• Employ knowledge distillation to transfer knowledge from a large, pre-trained teacher model to a smaller student model using only public training data.
• Introduce adaptive norm bounding to dynamically control the magnitude of gradients during distillation, improving training stability and accelerating convergence.
• Apply differential privacy by injecting calibrated noise into the distilled knowledge (e.g., feature maps or logits) to ensure (ε, δ)-differential privacy.
• Design a query sample selection mechanism that reduces the number of queries to the teacher model, thereby minimizing privacy budget consumption.
• Integrate self-learning during distillation to improve student model performance without additional queries or privacy cost.
• Use a combination of hint learning (intermediate layer supervision), distillation learning (output soft label matching), and self-learning (pseudo-labeling) to enhance student model generalization.

Experimental results

Research questions

• RQ1 Can knowledge distillation be made privately secure when the teacher model is trained on sensitive user data, without exposing the data or model to the public?
• RQ2 How can differential privacy be effectively applied to distilled knowledge (e.g., feature maps or logits) to ensure provable privacy guarantees?
• RQ3 To what extent can model compression be achieved while maintaining high accuracy and low privacy budget consumption?
• RQ4 Can adaptive query selection and self-learning techniques reduce the number of queries to the teacher model without degrading performance?
• RQ5 How does the framework perform under realistic conditions where certain data classes (e.g., digits 6 and 9) are considered highly sensitive and excluded from training?

Key findings

• On SVHN, Rona achieves a 20× compression ratio and 19× speed-up with only 0.97% accuracy loss while guaranteeing (9.83, 10⁻⁶)-differential privacy.
• On MNIST, Rona achieves 98.64% accuracy with (7.68, 10⁻⁵)-differential privacy, outperforming prior work under comparable privacy constraints.
• On CIFAR-10, Rona achieves 81.69% accuracy with (8.87, 10⁻⁵)-differential privacy, surpassing previous methods that reported only 73% accuracy under similar privacy budgets.
• The adaptive norm bound method significantly accelerates training and improves accuracy, achieving higher performance with fewer query epochs and lower privacy loss.
• Even when the student model has never seen certain sensitive classes (e.g., digits 6 and 9 on MNIST), Rona improves recognition accuracy from 0% to 53.43% under (ε=29.8, δ=10⁻⁶) privacy.
• The framework enables efficient on-device deployment: on a Huawei Honor 8, the student model runs 11× faster than the teacher model on MNIST with only 0.2% accuracy drop.