[Paper Review] Online Knowledge Distillation via Multi-branch Diversity Enhancement

Zheng Li, Ying Huang+4 more|arXiv (Cornell University)|Oct 2, 2020

Domain Adaptation and Few-Shot Learning50 references4 citations

TL;DR

This paper proposes a novel online knowledge distillation framework that enhances diversity among multiple student models via a Feature Fusion Module (FFM) and a Classifier Diversification (CD) loss. By fusing high-level semantic features from multiple branches and enforcing orthogonal feature learning, the method achieves state-of-the-art performance on CIFAR-10/100 and CINIC-10 with no additional inference cost.

ABSTRACT

Knowledge distillation is an effective method to transfer the knowledge from the cumbersome teacher model to the lightweight student model. Online knowledge distillation uses the ensembled prediction results of multiple student models as soft targets to train each student model. However, the homogenization problem will lead to difficulty in further improving model performance. In this work, we propose a new distillation method to enhance the diversity among multiple student models. We introduce Feature Fusion Module (FFM), which improves the performance of the attention mechanism in the network by integrating rich semantic information contained in the last block of multiple student models. Furthermore, we use the Classifier Diversification(CD) loss function to strengthen the differences between the student models and deliver a better ensemble result. Extensive experiments proved that our method significantly enhances the diversity among student models and brings better distillation performance. We evaluate our method on three image classification datasets: CIFAR-10/100 and CINIC-10. The results show that our method achieves state-of-the-art performance on these datasets.

Motivation & Objective

• To address the homogenization problem in online knowledge distillation, where multiple student models converge to similar representations, reducing distillation effectiveness.
• To improve the performance of ensemble-based student models by enhancing feature diversity across branches.
• To develop a lightweight, efficient method that maintains high accuracy while avoiding the need for a pre-trained teacher model.
• To achieve state-of-the-art performance on image classification benchmarks using only the group leader as the final deployed model.

Proposed method

• Introduces the Feature Fusion Module (FFM), which aggregates semantic features from the last block of multiple student models to enrich attention mechanism inputs.
• Employs a Classifier Diversification (CD) loss function that regularizes feature learning to enforce orthogonality between student branches, reducing homogenization.
• Adopts a two-level distillation framework with m-1 auxiliary branches and one group leader, where knowledge is distilled into the leader for deployment.
• Uses the ensemble predictions of auxiliary branches as soft targets for training the group leader, with FFM enhancing attention quality through multi-branch feature fusion.
• Applies the CD loss as a regularization term that penalizes similarity between branch features, promoting distinct learning patterns.
• Optimizes the hyperparameter γ in the CD loss to balance diversity and convergence, ensuring stable training.

Experimental results

Research questions

• RQ1 Can multi-branch feature fusion improve attention mechanism performance in online knowledge distillation?
• RQ2 Does enforcing orthogonal feature learning across student branches reduce model homogenization and improve ensemble accuracy?
• RQ3 Can the proposed method achieve state-of-the-art performance on image classification benchmarks without increasing inference cost?
• RQ4 How sensitive is the CD loss to hyperparameter γ, and what range ensures optimal performance?

Key findings

• The proposed method achieves a top-1 error rate of 24.84% on CIFAR-100 using ResNet-32, outperforming the baseline by 1.68 percentage points.
• On ResNet-110, the method reduces error by 2.13% compared to the baseline and 0.35% over the state-of-the-art OKDDip method.
• The combination of FFM and CD loss reduces top-1 error by 0.56% compared to FFM alone, demonstrating the effectiveness of both components.
• The CD loss is the most influential component, significantly improving performance across all backbone networks and datasets.
• The method maintains robust performance across a wide range of γ values, indicating stability in hyperparameter selection.
• On CINIC-10, the method achieves state-of-the-art results, confirming its generalization across different datasets.