[Paper Review] Improved Feature Distillation via Projector Ensemble

Yudong Chen, Sen Wang+4 more|arXiv (Cornell University)|Oct 27, 2022

Industrial Vision Systems and Defect Detection14 citations

TL;DR

This paper proposes a projector ensemble method to improve feature distillation in knowledge distillation, demonstrating that even with identical feature dimensions, a projector reduces overfitting to the teacher's feature space and enhances student performance. By ensembling multiple projectors with different random initializations, the method improves feature generalization and achieves state-of-the-art accuracy across diverse teacher-student pairs on ImageNet and CIFAR-100.

ABSTRACT

In knowledge distillation, previous feature distillation methods mainly focus on the design of loss functions and the selection of the distilled layers, while the effect of the feature projector between the student and the teacher remains under-explored. In this paper, we first discuss a plausible mechanism of the projector with empirical evidence and then propose a new feature distillation method based on a projector ensemble for further performance improvement. We observe that the student network benefits from a projector even if the feature dimensions of the student and the teacher are the same. Training a student backbone without a projector can be considered as a multi-task learning process, namely achieving discriminative feature extraction for classification and feature matching between the student and the teacher for distillation at the same time. We hypothesize and empirically verify that without a projector, the student network tends to overfit the teacher's feature distributions despite having different architecture and weights initialization. This leads to degradation on the quality of the student's deep features that are eventually used in classification. Adding a projector, on the other hand, disentangles the two learning tasks and helps the student network to focus better on the main feature extraction task while still being able to utilize teacher features as a guidance through the projector. Motivated by the positive effect of the projector in feature distillation, we propose an ensemble of projectors to further improve the quality of student features. Experimental results on different datasets with a series of teacher-student pairs illustrate the effectiveness of the proposed method.

Motivation & Objective

• To investigate why feature distillation improves even when student and teacher have identical feature dimensions, challenging the assumption that projectors are only needed for dimension mismatch.
• To address the overfitting issue in student networks that occurs when they simultaneously optimize for classification and feature matching with the teacher.
• To improve distillation performance by introducing an ensemble of projectors that diversify feature transformations and enhance generalization.
• To develop a simple yet effective method that outperforms state-of-the-art feature distillation techniques without adding complex auxiliary networks or increasing training cost significantly.

Proposed method

• The method introduces a projector between the student and teacher feature representations, even when dimensions are identical, to decouple classification and feature matching objectives.
• Multiple projectors with different random initializations are ensembled to generate diverse transformed features, leveraging principles from ensemble learning for improved robustness.
• The student network is trained using a combined loss: cross-entropy loss for classification and a direction alignment loss for feature distillation.
• The direction alignment loss minimizes the angular difference between student and teacher features after projection, promoting structural similarity without requiring identical dimensions.
• The framework is plug-and-play, requiring only the addition of projectors and no modification to the student or teacher architecture.
• The method is evaluated across multiple datasets and teacher-student pairs, including ResNet, MobileNet, and DenseNet combinations.

Experimental results

Research questions

• RQ1 Why does adding a projector improve distillation performance even when the student and teacher have the same feature dimensionality?
• RQ2 How does the absence of a projector lead to overfitting on the teacher's feature distribution, and what is the mechanism behind this degradation?
• RQ3 Can an ensemble of projectors with different initializations further improve feature generalization and distillation performance?
• RQ4 How does the proposed projector ensemble compare in performance and efficiency to existing state-of-the-art feature distillation methods?
• RQ5 Does using only a direction alignment loss in the projector ensemble maintain high performance while reducing computational complexity?

Key findings

• The proposed method achieves state-of-the-art top-1 accuracy on ImageNet across multiple teacher-student pairs, outperforming methods like CRD, SRRL, CID, AFD, and KR.
• With the ResNet34-MobileNet student-teacher pair, the method achieves 71.94% top-1 accuracy, surpassing AFD (71.38%) and KR (71.61%) on the same setup.
• On the ResNet50-MobileNet pair, the method achieves 73.16% top-1 accuracy, exceeding KR’s 72.56% and AFD’s 72.56% (though AFD results are not fully reported, the trend is consistent).
• The method converges faster than other distillation methods, as shown in training curves on ImageNet, indicating improved optimization dynamics.
• Despite using multiple projectors, the training time (2,995 seconds per epoch) and GPU memory usage (11,523 MB) are comparable to simpler methods like KD and SP, showing low computational overhead.
• The method consistently improves performance across diverse architectures, including ResNet34, ResNet50, and DenseNet201 as teachers, demonstrating broad applicability.