[Paper Review] Channel Distillation: Channel-Wise Attention for Knowledge Distillation

Zaida Zhou, Chaoran Zhuge+2 more|arXiv (Cornell University)|Jun 2, 2020

Advanced Neural Network Applications36 references40 citations

TL;DR

The paper introduces Channel Distillation (CD) to transfer channel-wise attention from teacher to student, augments it with Guided Knowledge Distillation (GKD) that only uses correctly-predicted teacher outputs, and applies an Early Decay Teacher (EDT) strategy to gradually reduce distillation influence, achieving state-of-the-art results on ImageNet and strong CIFAR100 gains.

ABSTRACT

Knowledge distillation is to transfer the knowledge from the data learned by the teacher network to the student network, so that the student has the advantage of less parameters and less calculations, and the accuracy is close to the teacher. In this paper, we propose a new distillation method, which contains two transfer distillation strategies and a loss decay strategy. The first transfer strategy is based on channel-wise attention, called Channel Distillation (CD). CD transfers the channel information from the teacher to the student. The second is Guided Knowledge Distillation (GKD). Unlike Knowledge Distillation (KD), which allows the student to mimic each sample's prediction distribution of the teacher, GKD only enables the student to mimic the correct output of the teacher. The last part is Early Decay Teacher (EDT). During the training process, we gradually decay the weight of the distillation loss. The purpose is to enable the student to gradually control the optimization rather than the teacher. Our proposed method is evaluated on ImageNet and CIFAR100. On ImageNet, we achieve 27.68% of top-1 error with ResNet18, which outperforms state-of-the-art methods. On CIFAR100, we achieve surprising result that the student outperforms the teacher. Code is available at https://github.com/zhouzaida/channel-distillation.

Motivation & Objective

• Motivate improving knowledge distillation by transferring channel-wise attention rather than raw representations.
• Propose Channel Distillation (CD) to transfer channel attention from teacher to student.
• Introduce Guided Knowledge Distillation (GKD) to use only correctly-predicted teacher outputs for guidance.
• Incorporate Early Decay Teacher (EDT) to progressively reduce distillation influence during training.

Proposed method

• Channel Distillation (CD): transfer channel attention by aligning channel weights between teacher and student using a w_c-based loss over feature maps.
• Guided Knowledge Distillation (GKD): apply KD loss only to samples correctly predicted by the teacher, ignoring incorrect teacher predictions.
• Early Decay Teacher (EDT): gradually decay the distillation loss weight as training progresses.
• Loss formulation: Loss(s,t) = EDT(alpha)CD(s,t) + GKD(s,t) + CE(s,y).
• Ablation and evaluation on ImageNet (ResNet34→ResNet18) and CIFAR100 (ResNet152→ResNet50) against KD, FitNets, AT, RKD, and more.

Experimental results

Research questions

• RQ1 Does channel-wise attention transfer via CD improve student performance over traditional KD?
• RQ2 Does restricting KD to correctly predicted teacher outputs via GKD reduce negative transfer?
• RQ3 Does progressively decaying distillation influence lead to better local optima and final accuracy?
• RQ4 How does the proposed CD+GKD+EDT compare to existing distillation methods on large-scale (ImageNet) and small-scale (CIFAR100) datasets?

Key findings

• On ImageNet, CD alone outperforms standard KD in top-1 error with ResNet18 (27.68% top-1 error reported in abstract).
• CD+GKD improves further to 27.61% top-1 error and 9.20% top-5 error.
• CD+GKD+EDT achieves state-of-the-art performance among listed methods on ImageNet with top-1/ top-5 errors of 27.61% and 9.20%, respectively.
• On CIFAR100, CD distillation allows a ResNet50 student to surpass the ResNet152 teacher under certain settings.
• GKD reduces negative transfer by only incorporating knowledge from teacher’s correct predictions.
• EDT gradually reduces distillation weight, enabling the student to refine its own optimization.