[Paper Review] MSD: Multi-Self-Distillation Learning via Multi-classifiers within Deep Neural Networks

Yunteng Luan, Hanyu Zhao+2 more|arXiv (Cornell University)|Nov 21, 2019

Anomaly Detection Techniques and Applications16 references17 citations

TL;DR

This paper proposes Multi-Self-Distillation (MSD), a training framework that enhances accuracy across multiple classifiers in deep neural networks by enabling them to collaboratively learn through knowledge distillation. By applying three losses—supervised, prediction mimicry, and feature mimicry—MSD improves performance on both multi-classifier networks (e.g., MSDNet) and single-classifier networks (e.g., ResNet) with added early branches, achieving up to 8.5% accuracy gain on first classifiers and over 1% average improvement across all classifiers.

ABSTRACT

As the development of neural networks, more and more deep neural networks are adopted in various tasks, such as image classification. However, as the huge computational overhead, these networks could not be applied on mobile devices or other low latency scenes. To address this dilemma, multi-classifier convolutional network is proposed to allow faster inference via early classifiers with the corresponding classifiers. These networks utilize sophisticated designing to increase the early classifier accuracy. However, naively training the multi-classifier network could hurt the performance (accuracy) of deep neural networks as early classifiers throughout interfere with the feature generation process. In this paper, we propose a general training framework named multi-self-distillation learning (MSD), which mining knowledge of different classifiers within the same network and increase every classifier accuracy. Our approach can be applied not only to multi-classifier networks, but also modern CNNs (e.g., ResNet Series) augmented with additional side branch classifiers. We use sampling-based branch augmentation technique to transform a single-classifier network into a multi-classifier network. This reduces the gap of capacity between different classifiers, and improves the effectiveness of applying MSD. Our experiments show that MSD improves the accuracy of various networks: enhancing the accuracy of every classifier significantly for existing multi-classifier network (MSDNet), improving vanilla single-classifier networks with internal classifiers with high accuracy, while also improving the final accuracy.

Motivation & Objective

• To address the performance degradation in multi-classifier networks caused by early classifiers interfering with feature learning.
• To enable knowledge transfer among multiple classifiers within a single network without requiring external student models.
• To enhance accuracy of both early and final classifiers in deep networks, especially in resource-constrained inference scenarios.
• To extend single-classifier networks into effective multi-classifier networks using a sampling-based branch augmentation technique.

Proposed method

• Introduce a multi-self-distillation (MSD) framework where all classifiers in a network act as student models, learning from each other via knowledge distillation.
• Train each classifier with three losses: cross-entropy (supervised), prediction mimicry (aligning class posteriors across classifiers), and feature mimicry (aligning feature maps with the deepest classifier).
• Apply feature mimicry loss to align features across heterogeneous classifiers (shallow and deep), promoting knowledge transfer and reducing bias toward deeper networks.
• Use a sampling-based branch augmentation method to insert early-classifier branches into single-classifier networks at strategic group boundaries, preserving capacity balance.
• Ensure dynamic inference by allowing the network to exit at any early classifier branch, enabling faster inference for easy samples.
• Maintain original network architecture and training hyperparameters, applying MSD as a plug-in training strategy.

Experimental results

Research questions

• RQ1 Can knowledge distillation among internal classifiers improve accuracy across all stages of a multi-classifier network?
• RQ2 Does collaborative learning among classifiers reduce interference and enhance feature representation quality?
• RQ3 Can MSD be effectively applied to single-classifier networks to create high-accuracy multi-classifier inference paths?
• RQ4 How does MSD compare to traditional self-distillation and joint training in terms of accuracy and training efficiency?
• RQ5 What is the impact of feature mimicry and prediction mimicry losses on classifier performance and generalization?

Key findings

• MSD improved the accuracy of every classifier in MSDNet by an average of over 1% on CIFAR100, with the first classifier gaining up to 1.73% over joint training.
• On ResNet18, the first classifier achieved 78.93% accuracy with MSD, a 1.84% improvement over naive training and 1.08% over self-distillation.
• For WRN20-8, the first classifier saw an 8.5% accuracy gain with MSD compared to self-distillation, demonstrating strong gains on early heads.
• The final classifier accuracy across all tested networks (ResNet18, ResNet50, etc.) improved by an average of 1% over self-distillation and 3.2% over naive training.
• The accuracy gap between the first and final classifiers was significantly reduced, indicating better knowledge transfer and balanced performance.
• MSD achieved these gains without increasing training cost or requiring external models, making it a plug-and-play training enhancement.