[Paper Review] Distilling Object Detectors with Task Adaptive Regularization

Ruoyu Sun, Fuhui Tang+3 more|arXiv (Cornell University)|Jun 23, 2020

Advanced Neural Network Applications31 references44 citations

TL;DR

The paper presents a task adaptive distillation framework for object detectors that uses region proposal sharing and module-specific imitation losses (backbone, classification head, and bounding box regression) plus a distillation decay strategy, achieving 74.5 mAP on VOC with a ResNet-101–50 student that surpasses the teacher.

ABSTRACT

Current state-of-the-art object detectors are at the expense of high computational costs and are hard to deploy to low-end devices. Knowledge distillation, which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the promising solutions for model miniaturization. In this paper, we investigate each module of a typical detector in depth, and propose a general distillation framework that adaptively transfers knowledge from teacher to student according to the task specific priors. The intuition is that simply distilling all information from teacher to student is not advisable, instead we should only borrow priors from the teacher model where the student cannot perform well. Towards this goal, we propose a region proposal sharing mechanism to interflow region responses between the teacher and student models. Based on this, we adaptively transfer knowledge at three levels, \emph{i.e.}, feature backbone, classification head, and bounding box regression head, according to which model performs more reasonably. Furthermore, considering that it would introduce optimization dilemma when minimizing distillation loss and detection loss simultaneously, we propose a distillation decay strategy to help improve model generalization via gradually reducing the distillation penalty. Experiments on widely used detection benchmarks demonstrate the effectiveness of our method. In particular, using Faster R-CNN with FPN as an instantiation, we achieve an accuracy of $39.0\%$ with Resnet-50 on COCO dataset, which surpasses the baseline $36.3\%$ by $2.7\%$ points, and even better than the teacher model with $38.5\%$ mAP.

Motivation & Objective

• Motivate model miniaturization for heavy object detectors without sacrificing performance.
• Develop a general distillation framework tailored to detector modules (backbone, classification head, regression head).
• Leverage region proposal sharing to compare teacher/student generalization across regions.
• Introduce a distillation decay strategy to improve training stability and generalization.
• Demonstrate effectiveness on standard detection benchmarks (VOC, COCO) across Faster R-CNN and RetinaNet backbones.

Proposed method

• Introduce region proposal sharing between teacher and student to transfer region-level responses.
• Backbone distillation uses a 2D Gaussian mask to emphasize foreground within ground-truth boxes.
• Classification head distillation uses soft labels from the teacher on positive proposals and hard labels from ground truth.
• Bounding box regression distillation uses adaptive targets from the teacher only when IoU criteria favor the teacher’s guidance.
• Combine distillation losses with detection losses as a multi-task objective and apply a distillation decay strategy to gradually reduce distillation pressure.
• Empirically validate across Faster R-CNN with FPN and RetinaNet on VOC and COCO and report mAP improvements.

Experimental results

Research questions

• RQ1 Can detector-specific knowledge distillation be made adaptive to the task and module (backbone, classification, regression)?
• RQ2 Does sharing region proposals enable better cross-model generalization and effective distillation?
• RQ3 Will a gradually reduced distillation penalty improve convergence and final detector performance?
• RQ4 How do Gaussian-based backbone distillations and adaptive head distillations compare to naïve feature imitation in object detection?

Key findings

• On VOC 2007 with R-101 (teacher) and R-50 (student), distillation yields mAP of 74.5, higher than the teacher’s 74.3.
• Backbone distillation with Gaussian masks provides about 2.4 percentage points gain in mAP.
• Classification head distillation and regression head distillation each contribute substantial gains (3.2 and 3.4 points respectively).
• Distillation decay further improves VOC performance from 73.8 to 74.5 mAP, surpassing the teacher.
• On COCO, distillation improves RetinaNet and Faster R-CNN baselines and can produce competitive or superior results to larger teachers (e.g., Retina-101 improves Retina-50 by 2.3 mAP; R-152-50 reaches 39.9 mAP, surpassing some larger teachers).