[Paper Review] Like What You Like: Knowledge Distill via Neuron Selectivity Transfer

Zehao Huang, Naiyan Wang|arXiv (Cornell University)|Jul 5, 2017

Domain Adaptation and Few-Shot Learning41 references341 citations

TL;DR

NST treats knowledge transfer as distribution alignment of neuron selectivity patterns between teacher and student, using MMD to improve compact models. It yields strong gains across CIFAR and ImageNet and complements other KT methods.

ABSTRACT

Despite deep neural networks have demonstrated extraordinary power in various applications, their superior performances are at expense of high storage and computational costs. Consequently, the acceleration and compression of neural networks have attracted much attention recently. Knowledge Transfer (KT), which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the popular solutions. In this paper, we propose a novel knowledge transfer method by treating it as a distribution matching problem. Particularly, we match the distributions of neuron selectivity patterns between teacher and student networks. To achieve this goal, we devise a new KT loss function by minimizing the Maximum Mean Discrepancy (MMD) metric between these distributions. Combined with the original loss function, our method can significantly improve the performance of student networks. We validate the effectiveness of our method across several datasets, and further combine it with other KT methods to explore the best possible results. Last but not least, we fine-tune the model to other tasks such as object detection. The results are also encouraging, which confirm the transferability of the learned features.

Motivation & Objective

• Motivate network acceleration and compression through a new knowledge transfer paradigm.
• Introduce Neuron Selectivity Transfer (NST) that matches distributions of neuron activations between teacher and student.
• Leverage Maximum Mean Discrepancy (MMD) as the core NST loss to align neuron selectivity patterns.
• Demonstrate NST effectiveness across CIFAR-10/100, ImageNet, and downstream tasks like object detection.
• Show that NST can complement existing KT methods for improved results.

Proposed method

• Treat neuron activations as distributions of selectivity patterns across spatial locations.
• Define NST loss as cross-entropy plus a scaled MMD term between teacher and student feature maps.
• Normalize feature maps before computing MMD to equalize sample scales.
• Explore three kernels for MMD: linear, polynomial (second-order with c=0), and Gaussian.
• Demonstrate that NST with polynomial kernel often yields best performance.

Experimental results

Research questions

• RQ1 Can aligning the distributions of neuron selectivity between teacher and student via MMD improve student network performance beyond direct feature map matching?
• RQ2 How does NST compare with existing KT methods (KD, FitNet, AT) on standard datasets (CIFAR, ImageNet) and downstream tasks?
• RQ3 Is NST complementary to other KT methods and beneficial when combined (e.g., KD+NST)?

Key findings

• NST improves student performance over the baseline across CIFAR-10 and CIFAR-100.
• NST with polynomial kernel often yields the best CIFAR results among NST variants (e.g., 4.39 for CIFAR-10 and 23.46 for CIFAR-100 with Inception-BN in Table 1).
• On ImageNet, NST with second-order polynomial kernel delivers 0.9% top-1 and 0.5% top-5 gains over the baseline student.
• Combining NST with KD provides the best overall gains in CIFAR and ImageNet (e.g., KD+NST* yields top results).
• NST also improves object detection performance in PASCAL VOC 2007 when used with Faster R-CNN, often outperforming other KT methods in this task.