[Paper Review] Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN

Can Peng, Kun Zhao+1 more|arXiv (Cornell University)|Mar 9, 2020

Domain Adaptation and Few-Shot Learning24 references6 citations

TL;DR

This paper proposes Faster ILOD, an end-to-end incremental object detection framework based on Faster R-CNN that uses multi-network adaptive distillation to mitigate catastrophic forgetting. By leveraging the RPN's robustness to missing annotations and distilling knowledge across multiple networks, it achieves 13× faster inference and up to 5.92% higher mean average precision than prior methods on PASCAL VOC and COCO datasets.

ABSTRACT

The human vision and perception system is inherently incremental where new knowledge is continually learned over time whilst existing knowledge is retained. On the other hand, deep learning networks are ill-equipped for incremental learning. When a well-trained network is adapted to new categories, its performance on the old categories will dramatically degrade. To address this problem, incremental learning methods have been explored which preserve the old knowledge of deep learning models. However, the state-of-the-art incremental object detector employs an external fixed region proposal method that increases overall computation time and reduces accuracy comparing to Region Proposal Network (RPN) based object detectors such as Faster RCNN. The purpose of this paper is to design an efficient end-to-end incremental object detector using knowledge distillation. We first evaluate and analyze the performance of the RPN-based detector with classic distillation on incremental detection tasks. Then, we introduce multi-network adaptive distillation that properly retains knowledge from the old categories when fine-tuning the model for new task. Experiments on the benchmark datasets, PASCAL VOC and COCO, demonstrate that the proposed incremental detector based on Faster RCNN is more accurate as well as being 13 times faster than the baseline detector.

Motivation & Objective

• Address catastrophic forgetting in object detectors during incremental learning, where new classes are added without access to old training data.
• Overcome the limitations of existing incremental detectors that rely on external, fixed region proposal networks, which slow down inference and reduce accuracy.
• Design an end-to-end, RPN-based incremental detection framework that preserves performance on old classes while learning new ones efficiently.
• Improve generalization and robustness in real-world scenarios where old data may be unavailable or re-annotation is impractical.
• Demonstrate that multi-network adaptive distillation effectively preserves knowledge from previous classes during incremental fine-tuning.

Proposed method

• Evaluate the RPN's inherent robustness to missing annotations for old classes in incremental detection, finding that its anchor selection mechanism tolerates partial labeling.
• Propose Faster ILOD, an end-to-end incremental learning framework built on Faster R-CNN, replacing the external proposal network with a trainable RPN.
• Implement multi-network adaptive distillation that transfers knowledge from multiple previous model versions to the current model during fine-tuning.
• Use knowledge distillation to transfer soft labels from older models to the current detector, preserving detection confidence and feature representations for old classes.
• Apply distillation losses not only between the current and previous model but across multiple prior models to stabilize knowledge retention.
• Fine-tune the entire Faster R-CNN network end-to-end using only new class annotations, while leveraging distilled knowledge to prevent forgetting.

Experimental results

Research questions

• RQ1 Can the RPN in Faster R-CNN tolerate missing annotations for old classes during incremental object detection?
• RQ2 How effective is multi-network adaptive distillation in preserving detection performance on previously learned classes during incremental learning?
• RQ3 Does an end-to-end RPN-based incremental detector outperform prior methods relying on fixed external region proposal networks in speed and accuracy?
• RQ4 How does the performance of the proposed method vary across different incremental learning sequences and class difficulty levels?
• RQ5 To what extent does the method reduce catastrophic forgetting while maintaining high accuracy on new classes?

Key findings

• The RPN in Faster R-CNN demonstrates inherent robustness to missing annotations for old classes due to its anchor-based proposal mechanism, allowing partial labeling during incremental training.
• Faster ILOD achieves 13× faster inference than ILOD (built on Fast R-CNN with external proposals), reducing average detection time from 1396.66 ms to 109.52 ms per image on VOC with ResNet-50.
• On PASCAL VOC, Faster ILOD achieves an average mAP gain of 5.78% over ILOD and 1.67% over ILOD applied to Faster R-CNN in multi-step incremental learning (add one class at a time).
• On COCO, Faster ILOD achieves a 5.92% mAP gain over ILOD and 2.74% over ILOD applied to Faster R-CNN under multi-step incremental detection (IoU=0.5), with further gains at higher IoU thresholds.
• The method shows consistent improvement across different learning sequences, with the highest gains observed when learning challenging categories like 'person' that co-occur frequently with other classes.
• The adaptive distillation mechanism is most effective when old data is scarce or annotations are missing, as evidenced by higher gains on categories with high missing annotation rates and large training sets.