[Paper Review] Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher

Seyed Iman Mirzadeh, Mehrdad Farajtabar+2 more|arXiv (Cornell University)|Feb 9, 2019

Advanced Neural Network ApplicationsComputer Science26 references128 citations

TL;DR

This paper proposes multi-step knowledge distillation using a teacher assistant to bridge the performance gap between large teacher networks and small student networks. By introducing an intermediate-sized teacher assistant, the method improves knowledge transfer, especially when the student-teacher size gap is large, achieving state-of-the-art accuracy on CIFAR-10 and CIFAR-100 with ResNet and plain CNN architectures.

ABSTRACT

Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too gigantic to be deployed on edge devices like smart-phones or embedded sensor nodes. There has been efforts to compress these networks, and a popular method is knowledge distillation, where a large (a.k.a. teacher) pre-trained network is used to train a smaller (a.k.a. student) network. However, in this paper, we show that the student network performance degrades when the gap between student and teacher is large. Given a fixed student network, one cannot employ an arbitrarily large teacher, or in other words, a teacher can effectively transfer its knowledge to students up to a certain size, not smaller. To alleviate this shortcoming, we introduce multi-step knowledge distillation which employs an intermediate-sized network (a.k.a. teacher assistant) to bridge the gap between the student and the teacher. We study the effect of teacher assistant size and extend the framework to multi-step distillation. Moreover, empirical and theoretical analysis are conducted to analyze the teacher assistant knowledge distillation framework. Extensive experiments on CIFAR-10 and CIFAR-100 datasets and plain CNN and ResNet architectures substantiate the effectiveness of our proposed approach.

Motivation & Objective

• Address the performance degradation in knowledge distillation when the student network is significantly smaller than the teacher network.
• Overcome the limitation that large teachers cannot effectively transfer knowledge to very small students due to a large architectural gap.
• Propose a multi-step distillation framework using an intermediate-sized teacher assistant to act as a knowledge bridge.
• Investigate the impact of teacher assistant size and extend the framework to multi-step distillation for improved performance.

Proposed method

• Introduce a teacher assistant—a model of intermediate size between the student and the original teacher—between the student and the teacher in the distillation pipeline.
• Use the teacher assistant to distill knowledge from the large teacher network and transfer it to the smaller student network in a two-step process.
• Apply knowledge distillation at both steps: first from teacher to teacher assistant, then from teacher assistant to student.
• Optimize the distillation process using soft labels and feature-level knowledge transfer, with cross-entropy and KL divergence as loss components.
• Systematically vary the size of the teacher assistant to analyze its impact on student performance.
• Extend the framework to multi-step distillation by cascading multiple intermediate models to progressively close the gap between student and teacher.

Experimental results

Research questions

• RQ1 Can a teacher assistant effectively bridge the performance gap between a large teacher and a small student in knowledge distillation?
• RQ2 How does the size of the teacher assistant affect the final student model accuracy?
• RQ3 Does multi-step distillation outperform single-step distillation when the student-teacher size gap is large?
• RQ4 What is the theoretical and empirical basis for improved knowledge transfer using an intermediate model?

Key findings

• The proposed teacher assistant framework significantly improves student network accuracy compared to standard knowledge distillation when the student-teacher size gap is large.
• Performance degrades when the teacher is too large relative to the student, confirming the existence of a practical upper bound on effective knowledge transfer.
• The optimal teacher assistant size lies between the student and the original teacher, with performance peaking at intermediate sizes.
• Multi-step distillation with multiple teacher assistants further improves accuracy on CIFAR-10 and CIFAR-100, especially with deeper architectures like ResNet.
• Empirical results show consistent improvements across both plain CNN and ResNet models on CIFAR-10 and CIFAR-100 datasets.
• Theoretical and empirical analysis confirms that the teacher assistant reduces distribution shift and improves feature alignment during knowledge transfer.