[Paper Review] Spirit Distillation: A Model Compression Method with Multi-domain Knowledge Transfer

Zhiyuan Wu, Yu Jiang+5 more|arXiv (Cornell University)|Apr 29, 2021

Domain Adaptation and Few-Shot Learning33 references4 citations

TL;DR

This paper proposes Spirit Distillation (SD) and Enhanced Spirit Distillation (ESD), a novel knowledge distillation framework that enables cross-domain knowledge transfer from a pre-trained teacher network to a compact student network, improving model compression and robustness in few-shot learning. By transferring general features from the teacher’s front-end and incorporating a proximity domain for richer feature learning, ESD boosts mIOU by 1.4% and high-precision accuracy by 8.2% with only 41.8% FLOPs.

ABSTRACT

Recent applications pose requirements of both cross-domain knowledge transfer and model compression to machine learning models due to insufficient training data and limited computational resources. In this paper, we propose a new knowledge distillation model, named Spirit Distillation (SD), which is a model compression method with multi-domain knowledge transfer. The compact student network mimics out a representation equivalent to the front part of the teacher network, through which the general knowledge can be transferred from the source domain (teacher) to the target domain (student). To further improve the robustness of the student, we extend SD to Enhanced Spirit Distillation (ESD) in exploiting a more comprehensive knowledge by introducing the proximity domain which is similar to the target domain for feature extraction. Results demonstrate that our method can boost mIOU and high-precision accuracy by 1.4% and 8.2% respectively with 78.2% segmentation variance, and can gain a precise compact network with only 41.8% FLOPs.

Motivation & Objective

• Address the dual challenge of model compression and few-shot learning under data scarcity and limited computational resources.
• Enable effective cross-domain knowledge transfer by leveraging a teacher network trained on a source domain to improve a student network in a target domain.
• Improve the robustness and generalization of compact student networks through enhanced feature learning using a proximity domain similar to the target domain.
• Develop a unified framework that combines knowledge distillation with multi-domain knowledge transfer for efficient, high-performance model compression.

Proposed method

• Propose Spirit Distillation (SD), where the student network learns a representation equivalent to the front part of the teacher network, transferring general knowledge from the source domain.
• Extend SD to Enhanced Spirit Distillation (ESD) by introducing a proximity domain—data similar to the target domain—for more comprehensive feature extraction.
• Use feature-based distillation to transfer intermediate representations from the teacher’s early layers to the student, minimizing FLOPs while preserving performance.
• Train the student network using a combination of knowledge from the source domain (via teacher) and the proximity domain, improving generalization and reducing variance.
• Optimize the training process by adjusting the ratio $ r $ of proximity domain data, allowing controlled trade-offs between robustness and performance.
• Apply knowledge distillation loss on intermediate features, not just logits, to preserve structural and semantic knowledge across domains.

Experimental results

Research questions

• RQ1 Can knowledge distillation be effectively extended to support cross-domain knowledge transfer while enabling model compression?
• RQ2 How does incorporating a proximity domain—similar to the target domain—improve the robustness and performance of a compact student network?
• RQ3 To what extent can Spirit Distillation (SD) and Enhanced Spirit Distillation (ESD) improve mIOU and high-precision accuracy under limited training data?
• RQ4 What is the impact of varying the proportion of proximity domain data ($ r $) on the student network’s segmentation variance and robustness?

Key findings

• Spirit Distillation (SD) improves mIOU by 1.4% and high-precision accuracy by 8.2% compared to standard training, with only 41.8% of the FLOPs.
• Enhanced Spirit Distillation (ESD) reduces segmentation variance by 21.8% and increases high-precision accuracy by 8.2% compared to standard training.
• With $ r = 0.5 $, ESD achieves 89.2% high-precision accuracy and 4.48 × 10⁻³ variance, demonstrating superior robustness.
• The use of the proximity domain significantly improves prediction consistency, reducing the proportion of very poor segmentation results.
• Visual comparisons show that ESD enables better shadow and road boundary segmentation than standard training or SD alone.
• The method maintains high performance even with limited target domain data, confirming its effectiveness in few-shot learning scenarios.