[Paper Review] Disentangle and Remerge: Interventional Knowledge Distillation for Few-Shot Object Detection from A Conditional Causal Perspective

Jiangmeng Li, Yanan Zhang+6 more|arXiv (Cornell University)|Aug 26, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes D&R (Disentangle and Remerge), a causal intervention-based knowledge distillation method for few-shot object detection that mitigates performance degradation caused by teacher model errors. By modeling knowledge distillation through a Structural Causal Model and applying backdoor adjustment, D&R improves semantic feature learning, achieving state-of-the-art performance on COCO and VOC benchmarks with significant gains in few-shot settings.

ABSTRACT

Few-shot learning models learn representations with limited human annotations, and such a learning paradigm demonstrates practicability in various tasks, e.g., image classification, object detection, etc. However, few-shot object detection methods suffer from an intrinsic defect that the limited training data makes the model cannot sufficiently explore semantic information. To tackle this, we introduce knowledge distillation to the few-shot object detection learning paradigm. We further run a motivating experiment, which demonstrates that in the process of knowledge distillation, the empirical error of the teacher model degenerates the prediction performance of the few-shot object detection model as the student. To understand the reasons behind this phenomenon, we revisit the learning paradigm of knowledge distillation on the few-shot object detection task from the causal theoretic standpoint, and accordingly, develop a Structural Causal Model. Following the theoretical guidance, we propose a backdoor adjustment-based knowledge distillation method for the few-shot object detection task, namely Disentangle and Remerge (D&R), to perform conditional causal intervention toward the corresponding Structural Causal Model. Empirically, the experiments on benchmarks demonstrate that D&R can yield significant performance boosts in few-shot object detection. Code is available at https://github.com/ZYN-1101/DandR.git.

Motivation & Objective

• To address the performance degradation in few-shot object detection when using knowledge distillation due to teacher model errors.
• To investigate the root cause of performance drops in knowledge distillation for few-shot detection from a causal inference perspective.
• To develop a causal model that captures the relationships between teacher predictions, student learning, and target labels.
• To design a backdoor adjustment-based distillation method that intervenes in the causal structure to improve student generalization.
• To validate the effectiveness and robustness of the proposed method across multiple benchmarks and teacher models.

Proposed method

• The authors construct a Structural Causal Model (SCM) to formalize the causal relationships between input data, teacher model outputs, and student predictions in few-shot object detection.
• They identify that empirical errors in the teacher model negatively affect student performance, violating standard backdoor criteria due to latent confounding.
• D&R applies backdoor adjustment to intervene on the causal graph, disentangling the true semantic knowledge from teacher errors during distillation.
• The method uses a text-based teacher (e.g., CLIP text encoder) to provide dense semantic supervision, improving feature learning efficiency.
• It introduces learnable hyperparameters α and β to balance the influence of different knowledge components in the distillation loss.
• The distillation loss is optimized with temperature scaling and weighted loss components, with ablation studies confirming optimal hyperparameter settings.

Experimental results

Research questions

• RQ1 Why does knowledge distillation sometimes degrade performance in few-shot object detection despite using powerful teacher models?
• RQ2 How do empirical errors in the teacher model propagate and degrade student model predictions?
• RQ3 Can a causal framework help identify and correct harmful knowledge transfer in knowledge distillation?
• RQ4 What modifications to the distillation process are needed to ensure only high-quality knowledge is transferred?
• RQ5 How does using a text-based teacher model compare to image-based or word-embedding teachers in terms of performance and efficiency?

Key findings

• D&R achieves state-of-the-art performance on the COCO dataset, with a 21.66% mAP on 30-shot detection, outperforming previous methods.
• On VOC Novel Set 1, D&R achieves 66.8% mAP in 10-shot detection, ranking second among all variants and significantly outperforming w/o KD.
• The use of CLIP(T) as the teacher model yields top-2 performance across all shot settings, demonstrating the effectiveness of text-based knowledge distillation.
• The method is robust to distillation loss weights, with optimal performance at weight 5 and strong results even at weight 10.
• D&R outperforms variants using CLIP(I+T), indicating that the image encoder adds minimal benefit relative to the increased computational cost.
• Ablation studies confirm that α=4 and β=0.5 yield the best performance, validating the importance of hyperparameter tuning in the intervention mechanism.