[Paper Review] Class-Incremental Few-Shot Object Detection

Pengyang Li, Yanan Li+2 more|arXiv (Cornell University)|May 17, 2021

Domain Adaptation and Few-Shot Learning22 references13 citations

TL;DR

This paper proposes LEAST, a novel method for class-incremental few-shot object detection that minimizes forgetting, reduces training resource consumption, and enhances transfer capability. By decoupling the feature extractor, applying knowledge distillation with low resource cost, and using clustering-based exemplar selection, LEAST achieves state-of-the-art performance with a harmonic mean AP of 18.2 and AR of 33.9 on COCO under few-shot incremental settings.

ABSTRACT

Conventional detection networks usually need abundant labeled training samples, while humans can learn new concepts incrementally with just a few examples. This paper focuses on a more challenging but realistic class-incremental few-shot object detection problem (iFSD). It aims to incrementally transfer the model for novel objects from only a few annotated samples without catastrophically forgetting the previously learned ones. To tackle this problem, we propose a novel method LEAST, which can transfer with Less forgetting, fEwer training resources, And Stronger Transfer capability. Specifically, we first present the transfer strategy to reduce unnecessary weight adaptation and improve the transfer capability for iFSD. On this basis, we then integrate the knowledge distillation technique using a less resource-consuming approach to alleviate forgetting and propose a novel clustering-based exemplar selection process to preserve more discriminative features previously learned. Being a generic and effective method, LEAST can largely improve the iFSD performance on various benchmarks.

Motivation & Objective

• Address the challenge of incremental object detection with only a few labeled examples per novel class, while preventing catastrophic forgetting of previously learned base classes.
• Overcome the limitations of existing methods that either forget base knowledge or require excessive training resources during incremental learning.
• Develop a generic and efficient framework that balances performance on both base and novel classes in sequential learning scenarios.
• Minimize unnecessary weight adaptation during transfer to improve transfer capability and reduce resource demands.
• Preserve discriminative features of base classes using a representative exemplar selection strategy to enhance knowledge retention.

Proposed method

• Decouple the class-sensitive object feature extractor from the full detector to reduce unnecessary weight adaptation and improve transfer capability during incremental learning.
• Integrate knowledge distillation using a lightweight, resource-efficient approach to preserve the pre-trained model's discriminative distribution and mitigate forgetting.
• Propose a clustering-based exemplar selection algorithm to select a small number of representative base-class features that capture intra-class variance and distribution modes.
• Apply the proposed transfer strategy in a sequential learning pipeline where novel classes are introduced one at a time with only a few annotated samples.
• Use distillation loss to align the student detector’s output with the teacher’s knowledge, ensuring base class performance is maintained during incremental updates.
• Combine the three components—transfer strategy, distillation, and exemplar selection—into a unified, generic framework applicable across different benchmarks and settings.

Experimental results

Research questions

• RQ1 How can we reduce catastrophic forgetting in few-shot object detection when learning novel classes incrementally with only a few labeled examples?
• RQ2 What transfer strategy minimizes unnecessary weight adaptation while maximizing transfer capability in incremental detection?
• RQ3 How can knowledge distillation be applied efficiently to preserve base-class knowledge without increasing training resource consumption?
• RQ4 What exemplar selection method best captures the distribution and intra-class variability of base classes using a minimal number of samples?
• RQ5 Can a unified framework combining these components outperform existing state-of-the-art methods in both few-shot and continual incremental learning settings?

Key findings

• LEAST achieves a harmonic mean AP of 18.2 and AR of 33.9 on COCO with 10-shot learning, significantly outperforming prior methods including TFA and ONCE.
• The proposed transfer strategy (FIT_CSE) improves performance by 1.8 AP and 1.9 AR over the baseline FIX_ALL, demonstrating reduced weight adaptation and stronger transfer capability.
• Knowledge distillation (with loss denoted as $d$) increases novel class AP by 2.4 points compared to no distillation, showing effective forgetting mitigation.
• The clustering-based exemplar selection ($e$) improves harmonic mean AP by 1.6 points over random selection ($e_r$) and 0.8 points over class-average selection ($e_a$), proving better representation of base-class features.
• In continual learning, LEAST maintains stable performance across 20 incremental stages, while LEAST-NE (without exemplars) degrades to AP 12.1 and AR 26.5, confirming the necessity of exemplar selection.
• On cross-domain transfer from MS-COCO to Pascal VOC, LEAST achieves AP 18.2 and AR 33.9, significantly outperforming competitors, validating its generalization and robustness.