[Paper Review] Long-Tailed Visual Recognition via Self-Heterogeneous Integration with Knowledge Excavation

Yan Jin, Mengke Li+3 more|arXiv (Cornell University)|Apr 3, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes SHIKE, a novel mixture-of-experts (MoE) framework for long-tailed visual recognition that integrates depth-wise knowledge fusion (DKF) and dynamic knowledge transfer (DKT) to improve representation learning across head, medium, and tail classes. By fusing features from different network depths and suppressing the hardest negative samples during distillation, SHIKE achieves state-of-the-art accuracy of 56.3% on CIFAR100-LT, 60.3% on ImageNet-LT, 75.4% on iNaturalist 2018, and 41.9% on Places-LT.

ABSTRACT

Deep neural networks have made huge progress in the last few decades. However, as the real-world data often exhibits a long-tailed distribution, vanilla deep models tend to be heavily biased toward the majority classes. To address this problem, state-of-the-art methods usually adopt a mixture of experts (MoE) to focus on different parts of the long-tailed distribution. Experts in these methods are with the same model depth, which neglects the fact that different classes may have different preferences to be fit by models with different depths. To this end, we propose a novel MoE-based method called Self-Heterogeneous Integration with Knowledge Excavation (SHIKE). We first propose Depth-wise Knowledge Fusion (DKF) to fuse features between different shallow parts and the deep part in one network for each expert, which makes experts more diverse in terms of representation. Based on DKF, we further propose Dynamic Knowledge Transfer (DKT) to reduce the influence of the hardest negative class that has a non-negligible impact on the tail classes in our MoE framework. As a result, the classification accuracy of long-tailed data can be significantly improved, especially for the tail classes. SHIKE achieves the state-of-the-art performance of 56.3%, 60.3%, 75.4%, and 41.9% on CIFAR100-LT (IF100), ImageNet-LT, iNaturalist 2018, and Places-LT, respectively.

Motivation & Objective

• Address the bias of deep neural networks toward head classes in long-tailed data distributions.
• Overcome the limitation of existing MoE methods that use homogeneous experts with identical depth, which fails to capture class-specific depth preferences.
• Improve representation learning for tail classes by integrating knowledge from shallow and deep network parts through self-heterogeneous integration.
• Reduce the negative impact of the hardest negative samples during knowledge distillation in MoE frameworks.

Proposed method

• Propose Depth-wise Knowledge Fusion (DKF), a component that fuses intermediate features from shallow and deep parts of a network into high-level features for each expert, enhancing feature diversity and representation capacity.
• Introduce Dynamic Knowledge Transfer (DKT), which identifies and reweights non-target logits with large values from diverse experts to form a unified 'grand teacher' for distilling non-target predictions.
• Apply knowledge distillation using both target and non-target knowledge: the former via standard distillation loss, and the latter via DKT to suppress the hardest negative class influence.
• Train the MoE model with balanced softmax cross-entropy (BSCE) and jointly optimize DKF and DKT components to improve generalization across long-tailed classes.
• Design a heterogeneous expert architecture where each expert uses features from different depth levels (shallow, middle, deep), enabling experts to specialize in different parts of the long-tailed distribution.
• Utilize self-distillation to transfer knowledge from shallow network branches to deep experts, enabling self-adaptive learning regardless of sample quantity.

Experimental results

Research questions

• RQ1 Can integrating features from different network depths improve representation diversity and performance in long-tailed visual recognition?
• RQ2 Does the preference of deep networks for certain classes vary with depth, and can this be exploited to improve tail-class generalization?
• RQ3 Can dynamic knowledge transfer from diverse experts effectively suppress the influence of the hardest negative samples during distillation?
• RQ4 How does the number and heterogeneity of experts affect performance in MoE-based long-tailed learning?
• RQ5 Can self-heterogeneous integration of knowledge from shallow and deep features lead to better generalization than standard MoE with homogeneous experts?

Key findings

• SHIKE achieves state-of-the-art performance with 56.3% accuracy on CIFAR100-LT (IF100), significantly outperforming baseline MoE models.
• On ImageNet-LT, SHIKE achieves 60.3% accuracy, demonstrating strong generalization on large-scale long-tailed data.
• On iNaturalist 2018, SHIKE reaches 75.4% top-1 accuracy, indicating robustness on fine-grained, long-tailed benchmarks.
• On Places-LT, SHIKE attains 41.9% accuracy, showing effectiveness on scene recognition with extreme class imbalance.
• Ablation studies confirm that DKF is the foundational component, contributing 0.59% accuracy gain when other components are excluded.
• DKT reduces the number of high-probability hardest negative samples by 30% on CIFAR100-LT, validating its effectiveness in suppressing negative interference during distillation.