[Paper Review] Enabling Deep Learning on Edge Devices through Filter Pruning and Knowledge Transfer

Kaiqi Zhao, Yitao Chen+1 more|arXiv (Cornell University)|Jan 22, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a novel framework combining filter pruning and knowledge transfer to enable efficient, accurate, and incremental on-device training of deep learning models on resource-constrained edge devices. By jointly compressing models into thinner, shallower architectures and transferring problem-solving knowledge from a cloud-based teacher model, the method achieves real-time convergence (3–6 minutes), preserves over 90% accuracy on old categories, and attains 78.92% Top-1 accuracy on unseen categories without retraining.

ABSTRACT

Deep learning models have introduced various intelligent applications to edge devices, such as image classification, speech recognition, and augmented reality. There is an increasing need of training such models on the devices in order to deliver personalized, responsive, and private learning. To address this need, this paper presents a new solution for deploying and training state-of-the-art models on the resource-constrained devices. First, the paper proposes a novel filter-pruning-based model compression method to create lightweight trainable models from large models trained in the cloud, without much loss of accuracy. Second, it proposes a novel knowledge transfer method to enable the on-device model to update incrementally in real time or near real time using incremental learning on new data and enable the on-device model to learn the unseen categories with the help of the in-cloud model in an unsupervised fashion. The results show that 1) our model compression method can remove up to 99.36% parameters of WRN-28-10, while preserving a Top-1 accuracy of over 90% on CIFAR-10; 2) our knowledge transfer method enables the compressed models to achieve more than 90% accuracy on CIFAR-10 and retain good accuracy on old categories; 3) it allows the compressed models to converge within real time (three to six minutes) on the edge for incremental learning tasks; 4) it enables the model to classify unseen categories of data (78.92% Top-1 accuracy) that it is never trained with.

Motivation & Objective

• To address the challenge of deploying and training large deep learning models on memory- and compute-constrained edge devices.
• To enable on-device model training that is personalized, responsive, and privacy-preserving by learning from local data in real time.
• To overcome the limitations of existing model compression and knowledge distillation techniques, which often fail to support incremental learning or generalize to unseen categories.
• To develop a compressed model that is both trainable and suitable for knowledge transfer, ensuring minimal accuracy degradation after compression.
• To enable on-device models to learn new categories without forgetting old ones, even without access to original training data or labels.

Proposed method

• A filter-pruning-based model compression method removes redundant filters with weak activation patterns and similar convolutional layers from a pre-trained cloud model, reducing model depth and width.
• The compressed model retains the same architecture as the original, enabling effective knowledge transfer between the teacher (cloud) and student (edge) models.
• A novel knowledge transfer method transfers intermediate representations and problem-solving processes from the teacher to the student, rather than just output logits or feature maps.
• The method selects optimal teacher-student layer pairs to maximize knowledge transfer efficiency and mitigate catastrophic forgetting during incremental learning.
• The student model is trained incrementally on new local data using distilled knowledge, enabling real-time adaptation without retraining on old data.
• The framework supports unsupervised learning of unseen categories by leveraging the teacher model’s knowledge, even when the student has never seen those classes before.

Experimental results

Research questions

• RQ1 Can a model compression technique reduce model size significantly while preserving high accuracy for edge deployment?
• RQ2 Can the compressed model be fine-tuned incrementally on-device with minimal latency and without catastrophic forgetting?
• RQ3 Can knowledge transfer from a cloud-based teacher model enable the on-device student to classify unseen categories accurately?
• RQ4 Does transferring the problem-solving process (via intermediate representations) outperform traditional knowledge distillation that only transfers soft labels or feature maps?
• RQ5 Can the entire pipeline achieve real-time convergence (within minutes) on edge hardware for incremental learning tasks?

Key findings

• The proposed model compression method reduces 99.36% of the parameters in WRN-28-10 while maintaining over 90% Top-1 accuracy on CIFAR-10.
• The method achieves a compression ratio of up to 139X on VGG-16 with less than 10% accuracy loss on Caltech 101.
• The compressed models converge within 3 to 6 minutes on edge devices for incremental learning tasks, enabling real-time adaptation.
• The knowledge transfer method enables the student model to achieve 78.92% Top-1 accuracy on unseen categories in CIFAR-10, despite never being trained on them.
• The model retains strong performance on old categories (over 90% accuracy) during incremental learning, demonstrating effective mitigation of catastrophic forgetting.
• Transferring the problem-solving process via intermediate representations significantly outperforms baseline knowledge distillation, achieving 78.92% accuracy on unseen classes compared to 0.00% with FitNet.