[论文解读] Structural Knowledge Distillation for Object Detection

Philip de Rijk, Lukas Schneider+2 more|arXiv (Cornell University)|Nov 23, 2022

Advanced Neural Network Applications被引用 21

一句话总结

论文用基于 SSIM 的损失替代像素级的 Lp 特征蒸馏，以捕捉亮度、对比度和结构，带来 RetinaNet 和 Faster R-CNN 在 MSCOCO 上的一致 AP 提升，且常常超过现有的 KD 方法。

ABSTRACT

Knowledge Distillation (KD) is a well-known training paradigm in deep neural networks where knowledge acquired by a large teacher model is transferred to a small student. KD has proven to be an effective technique to significantly improve the student's performance for various tasks including object detection. As such, KD techniques mostly rely on guidance at the intermediate feature level, which is typically implemented by minimizing an lp-norm distance between teacher and student activations during training. In this paper, we propose a replacement for the pixel-wise independent lp-norm based on the structural similarity (SSIM). By taking into account additional contrast and structural cues, feature importance, correlation and spatial dependence in the feature space are considered in the loss formulation. Extensive experiments on MSCOCO demonstrate the effectiveness of our method across different training schemes and architectures. Our method adds only little computational overhead, is straightforward to implement and at the same time it significantly outperforms the standard lp-norms. Moreover, more complex state-of-the-art KD methods using attention-based sampling mechanisms are outperformed, including a +3.5 AP gain using a Faster R-CNN R-50 compared to a vanilla model.

研究动机与目标

• Motivate reducing knowledge gap between large teachers and compact students in object detection without complex sampling schemes.
• Introduce a feature-based distillation loss based on SSIM to capture local mean, variance, and cross-correlation between teacher and student features.
• Demonstrate that SSIM-based KD yields superior detection performance across multiple architectures and training setups on MSCOCO.
• Show that the proposed method is lightweight (one-line code change) and can outperform state-of-the-art KD methods relying on attention-based sampling.

提出的方法

• Replace the conventional Lp feature distillation with an SSIM-based loss capturing local luminance (mean), contrast (variance), and structure (cross-correlation).
• Compute three SSIM components (luminance, contrast, structure) over 11x11 Gaussian patches to form a combined loss L_ssim.
• Normalize and optionally adapt teacher/student features (min-max normalization and a 1x1 conv) before applying the distillation loss.
• Combine distillation loss with the original detection loss as L = lambda * L_feat + L_det, where lambda is a tunable weight.
• Evaluate on MSCOCO using RetinaNet and Faster R-CNN with ResNet/ResNeXt backbones, training in PyTorch/MMDetection2.
• Explore ablations including influence of luminance/contrast/structure (alpha, beta, gamma), patch size, and presence of adaptation layers.

实验结果

研究问题

• RQ1 Does SSIM-based distillation outperform traditional Lp-based feature distillation for object detectors?
• RQ2 How do luminance, contrast, and structure components contribute to knowledge transfer and detection performance?
• RQ3 Is the SSIM-based KD robust across different detector architectures and training schedules?
• RQ4 Can simple one-line code changes implement effective KD without complex sampling mechanisms?
• RQ5 How does SSIM-based KD compare to state-of-the-art KD methods that rely on attention-based sampling?

主要发现

Backbone / Method	AP	AP50	AP75	AP_S	AP_M	AP_L
RetinaNet-R50 (Ours, SSIM)	40.1	59.2	43.1	23.1	44.6	53.2
RetinaNet-R50 (L2)	36.8	55.7	39.1	20.6	40.5	47.3
RetinaNet-R50 (L1)	38.7	57.6	41.6	22.7	42.7	50.5
Faster R-CNN-R50 (Ours, SSIM)	40.9	61.0	44.9	23.7	44.5	53.5
Faster R-CNN-R50 (L2)	37.4	57.6	40.9	21.2	41.3	48.1
Faster R-CNN-R50 (L1)	38.6	58.8	42.1	21.8	42.1	49.9

• SSIM-based distillation outperforms Lp norms by up to 3.7 AP on MSCOCO across RetinaNet and Faster R-CNN.
• Using SSIM yields more distributed error signaling across the feature space, guiding the student toward the teacher more effectively.
• The structure component (gamma) has the strongest positive impact, with gamma-only configurations achieving up to +3.2 AP.
• Integrating SSIM-based KD with various backbones and detectors consistently improves AP across S, M, and L object sizes.
• Our method matches or surpasses state-of-the-art KD methods (e.g., Zhang and Ma; Kang et al.) in AP gains, often with better large-object performance (AP_L).
• Adaptation layers are beneficial when teacher and student architectures differ; otherwise, they may be optional.
• A single line change (replacing L2 with L_ssim) suffices to deploy the method in existing pipelines.