[Paper Review] SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection

Junsu Kim, Hoseong Cho+3 more|arXiv (Cornell University)|Feb 27, 2024

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes SDDGR, a novel stable diffusion-based deep generative replay method for class incremental object detection (CIOD) that generates high-fidelity synthetic images of old classes using a pre-trained text-to-image diffusion model. By combining iterative refinement, L2 knowledge distillation, and pseudo-labeling for old objects in new-task images, SDDGR achieves state-of-the-art performance on COCO 2017, attaining 40.9% AP on old categories in the 70+10 scenario.

ABSTRACT

In the field of class incremental learning (CIL), generative replay has become increasingly prominent as a method to mitigate the catastrophic forgetting, alongside the continuous improvements in generative models. However, its application in class incremental object detection (CIOD) has been significantly limited, primarily due to the complexities of scenes involving multiple labels. In this paper, we propose a novel approach called stable diffusion deep generative replay (SDDGR) for CIOD. Our method utilizes a diffusion-based generative model with pre-trained text-to-diffusion networks to generate realistic and diverse synthetic images. SDDGR incorporates an iterative refinement strategy to produce high-quality images encompassing old classes. Additionally, we adopt an L2 knowledge distillation technique to improve the retention of prior knowledge in synthetic images. Furthermore, our approach includes pseudo-labeling for old objects within new task images, preventing misclassification as background elements. Extensive experiments on the COCO 2017 dataset demonstrate that SDDGR significantly outperforms existing algorithms, achieving a new state-of-the-art in various CIOD scenarios. The source code will be made available to the public.

Motivation & Objective

• To address catastrophic forgetting in class incremental object detection (CIOD), where models forget previously learned classes when learning new ones.
• To overcome the limitations of existing generative replay methods in multi-object, multi-label scenes common in object detection.
• To adapt pre-trained stable diffusion models—originally designed for image generation—for high-quality, class-specific synthetic data generation in CIOD.
• To improve knowledge retention in incremental learning by combining synthetic data generation with L2 knowledge distillation and pseudo-labeling.
• To demonstrate that diffusion-based generative replay can outperform existing methods in CIOD, particularly in complex, multi-object scenarios.

Proposed method

• Utilizes a pre-trained text-to-image diffusion model (Stable Diffusion) to generate synthetic images containing objects from previously learned classes using class-specific prompts and grounding inputs.
• Applies an iterative refinement strategy where a trained object detector evaluates and filters generated images based on detection confidence, retaining only high-fidelity samples.
• Implements class-wise regulation of synthetic image generation (N per class) to balance quality, diversity, and computational cost, avoiding over-generation of low-quality samples.
• Employs L2 knowledge distillation between the student model and the teacher model (trained on synthetic data) to transfer knowledge from synthetic images without direct end-to-end fine-tuning.
• Introduces pseudo-labeling for old-class objects detected in new-task images, ensuring they are not misclassified as background during training.
• Uses a dynamic refinement threshold (0.4–0.8) to balance image quality and quantity, with ablation showing fixed thresholds degrade performance.

Experimental results

Research questions

• RQ1 Can pre-trained stable diffusion models be effectively adapted for generating high-fidelity, class-specific synthetic images in multi-object detection scenarios?
• RQ2 How does iterative refinement of generated images impact the quality and utility of synthetic data in CIOD?
• RQ3 What is the optimal role of knowledge distillation when training on synthetic data in a class-incremental setting?
• RQ4 To what extent does pseudo-labeling old objects in new-task images reduce misclassification as background?
• RQ5 How do hyperparameters such as image generation count per class and refinement threshold affect overall performance?

Key findings

• SDDGR achieves a new state-of-the-art AP of 40.9% on old categories in the 70+10 CIOD scenario on COCO 2017, surpassing prior methods.
• Pseudo-labeling reduces false positive predictions (FPP) by 39.1% in AP, significantly improving detection of previously learned classes.
• The combination of synthetic data and L2 distillation increases AP by 1.2% across all categories and reduces FPP by 1.5% compared to baseline fine-tuning.
• Knowledge distillation with a weight λ=2 yields the best performance (AP=40.9%), outperforming all tested λ values and the prior SoTA of 40.4%.
• Using a dynamic refinement threshold range (0.4–0.8) improves performance by 1% compared to fixed thresholds, highlighting the importance of adaptive filtering.
• Limiting synthetic image generation to 100 per class (N=100) maintains high performance while reducing generation time, demonstrating scalability and efficiency.