[Paper Review] Understanding the Role of the Projector in Knowledge Distillation

Roy Miles, Krystian Mikolajczyk|arXiv (Cornell University)|Mar 20, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper rethinks knowledge distillation by identifying the projector as a critical component that implicitly encodes relational information across examples, enabling effective student training. By combining a linear projector, batch normalization, and a LogSum distance, the authors achieve state-of-the-art or superior performance on image classification, object detection, and data-efficient transformer training—especially attaining 77.2% top-1 accuracy on ImageNet with DeiT-Ti—while reducing computational and memory costs significantly.

ABSTRACT

In this paper we revisit the efficacy of knowledge distillation as a function matching and metric learning problem. In doing so we verify three important design decisions, namely the normalisation, soft maximum function, and projection layers as key ingredients. We theoretically show that the projector implicitly encodes information on past examples, enabling relational gradients for the student. We then show that the normalisation of representations is tightly coupled with the training dynamics of this projector, which can have a large impact on the students performance. Finally, we show that a simple soft maximum function can be used to address any significant capacity gap problems. Experimental results on various benchmark datasets demonstrate that using these insights can lead to superior or comparable performance to state-of-the-art knowledge distillation techniques, despite being much more computationally efficient. In particular, we obtain these results across image classification (CIFAR100 and ImageNet), object detection (COCO2017), and on more difficult distillation objectives, such as training data efficient transformers, whereby we attain a 77.2% top-1 accuracy with DeiT-Ti on ImageNet. Code and models are publicly available.

Motivation & Objective

• To understand the theoretical and practical role of the projector in knowledge distillation beyond mere feature transformation.
• To investigate how normalization and distance metrics interact with the projector to influence training dynamics and model performance.
• To develop a computationally efficient distillation recipe that avoids expensive relational structures like correlation matrices or memory banks.
• To evaluate the method across diverse tasks, including image classification, object detection, and data-efficient training of vision transformers.
• To demonstrate that simple architectural choices can outperform complex, specialized distillation methods, especially in high-capacity gap scenarios.

Proposed method

• The method employs a linear projection layer to transform teacher features before applying a LogSum distance to student features.
• Batch normalization is applied to the projected features to stabilize training dynamics and improve generalization.
• A LogSum distance function is used as the loss metric, which acts as a differentiable approximation to the soft maximum and improves performance on large capacity gap tasks.
• The projector implicitly captures relational gradients by encoding information from past examples, eliminating the need for explicit memory banks or correlation matrices.
• The framework is applied end-to-end to image classification (CIFAR100, ImageNet), object detection (COCO2017), and data-efficient training of vision transformers.
• The approach is designed to be plug-and-play, requiring no additional trainable layers or complex scheduling, and is compatible with existing training pipelines.

Experimental results

Research questions

• RQ1 How does the projector in knowledge distillation implicitly encode relational information between examples during training?
• RQ2 What is the role of normalization in coupling with the projector to influence training dynamics and final performance?
• RQ3 Can a simple LogSum distance function effectively close large capacity gaps between teacher and student models?
• RQ4 To what extent can a minimal set of components—linear projector, batch norm, LogSum loss—achieve performance comparable or superior to state-of-the-art distillation methods?
• RQ5 Does this distilled knowledge preserve explicit inductive biases, such as translational equivariance, from the teacher to the student?

Key findings

• The projector implicitly encodes relational gradients from past examples, enabling effective knowledge transfer without explicit memory or correlation matrices.
• Normalization of representations is tightly coupled with projector training dynamics and significantly impacts student performance.
• The LogSum distance function effectively addresses large capacity gap problems, improving performance on challenging distillation tasks.
• The proposed method achieves 77.2% top-1 accuracy on ImageNet using DeiT-Ti, outperforming specialized distillation methods for data-efficient training.
• The method improves translational equivariance transfer in vision transformers, with a measure of 0.04±0.02 compared to 1.52±0.15 for the teacher, indicating strong preservation of spatial inductive bias.
• On COCO2017 object detection, the method achieves 32.92 mAP (50-95) and 52.96 AP50, outperforming FPGI and matching or exceeding ReviewKD despite being significantly simpler and cheaper to implement.