[Paper Review] Embedded Self-Distillation in Compact Multi-Branch Ensemble Network for Remote Sensing Scene Classification

Qi Zhao, Yujing Ma+2 more|arXiv (Cornell University)|Apr 1, 2021

Remote-Sensing Image ClassificationEngineering65 references29 citations

TL;DR

This paper proposes ESD-MBENet, a compact multi-branch ensemble network with embedded self-distillation for remote sensing scene classification. By fusing logits and intermediate features across branches and transferring knowledge via self-distillation to a single main branch, the method achieves state-of-the-art accuracy on AID, NWPU-RESISC45, and UC-Merced datasets while enabling efficient inference using only the distilled main branch.

ABSTRACT

Remote sensing (RS) image scene classification task faces many challenges due to the interference from different characteristics of different geographical elements. To solve this problem, we propose a multi-branch ensemble network to enhance the feature representation ability by fusing features in final output logits and intermediate feature maps. However, simply adding branches will increase the complexity of models and decline the inference efficiency. On this issue, we embed self-distillation (SD) method to transfer knowledge from ensemble network to main-branch in it. Through optimizing with SD, main-branch will have close performance as ensemble network. During inference, we can cut other branches to simplify the whole model. In this paper, we first design compact multi-branch ensemble network, which can be trained in an end-to-end manner. Then, we insert SD method on output logits and feature maps. Compared to previous methods, our proposed architecture (ESD-MBENet) performs strongly on classification accuracy with compact design. Extensive experiments are applied on three benchmark RS datasets AID, NWPU-RESISC45 and UC-Merced with three classic baseline models, VGG16, ResNet50 and DenseNet121. Results prove that our proposed ESD-MBENet can achieve better accuracy than previous state-of-the-art (SOTA) complex models. Moreover, abundant visualization analysis make our method more convincing and interpretable.

Motivation & Objective

• Address the challenge of classifying remote sensing scenes with diverse geographical elements that cause interference in feature representation.
• Overcome the high computational and memory cost of multi-branch networks in real-world deployment, especially on embedded devices.
• Enhance feature representation by fusing multi-level features and logits from multiple branches in an end-to-end trainable architecture.
• Enable efficient inference by distilling knowledge from the full ensemble network into a single main branch, allowing pruning of sub-branches post-training.
• Achieve state-of-the-art performance with a compact, deployable model architecture.

Proposed method

• Design a compact multi-branch ensemble network (ESD-MBENet-v1 and v2) using weight-sharing blocks to generate multiple paths with minimal parameter increase.
• Introduce a zigzag branching structure in ESD-MBENet-v1 to generate diverse output logits from different combinations of shared blocks.
• Propose ESD-MBENet-v2 to enable flexible branch creation independent of split point depth, maximizing weight sharing and structural efficiency.
• Embed self-distillation (SD) on both final logits and intermediate feature maps, where the ensemble network acts as a teacher to guide the main-branch student network.
• Use soft-label knowledge distillation on logits and feature map distillation to transfer rich representations from the ensemble to the main branch.
• After training with SD, prune all sub-branches and deploy only the main branch for inference, maintaining high accuracy with reduced complexity.

Experimental results

Research questions

• RQ1 Can a compact multi-branch ensemble network with shared weights improve feature representation for remote sensing scene classification?
• RQ2 Does self-distillation on both logits and intermediate feature maps effectively transfer knowledge from a multi-branch ensemble to a single main branch?
• RQ3 Can the distilled main branch achieve performance comparable to the full ensemble model while significantly reducing inference complexity?
• RQ4 How does the proposed method compare to state-of-the-art models in terms of accuracy and efficiency on benchmark remote sensing datasets?
• RQ5 Is the proposed method generalizable to natural scene classification tasks beyond remote sensing?

Key findings

• ESD-MBENet achieves state-of-the-art classification accuracy on three benchmark remote sensing datasets: AID, NWPU-RESISC45, and UC-Merced, surpassing previous SOTA models.
• On the AID dataset with ResNet50 backbone, ESD-MBENet-v2 achieved 95.78% accuracy, outperforming the baseline and prior SOTA methods.
• On NWPU-RESISC45 with DenseNet121, ESD-MBENet-v2 reached 95.42% accuracy, demonstrating strong generalization across different backbones.
• The ablation study confirmed that joint distillation on both logits and feature maps yields better performance than distilling logits alone.
• Visualization via T-SNE and Grad-CAM showed that ESD-MBENet learns more discriminative and focused representations than the baseline, especially in deeper layers.
• Despite the ensemble structure, the final inference model is a single main branch, reducing FLOPs and parameters while maintaining high accuracy.