[Paper Review] Revisiting Few-Shot Object Detection with Vision-Language Models

Anish Madan, Neehar Peri+2 more|arXiv (Cornell University)|Dec 22, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Foundational Few-Shot Object Detection (Foundational FSOD), a new benchmark protocol that evaluates vision-language models (VLMs) by fine-tuning them on K-shot visual and textual examples of target concepts, rather than relying on pre-defined base/novel splits. It demonstrates that zero-shot VLMs like GroundingDINO outperform state-of-the-art FSOD methods (48.3 vs. 33.1 AP on COCO), and further improves performance via federated losses and multi-modal fine-tuning, achieving 5.9 AP gain on LVIS and nuImages.

ABSTRACT

The era of vision-language models (VLMs) trained on web-scale datasets challenges conventional formulations of "open-world" perception. In this work, we revisit the task of few-shot object detection (FSOD) in the context of recent foundational VLMs. First, we point out that zero-shot predictions from VLMs such as GroundingDINO significantly outperform state-of-the-art few-shot detectors (48 vs. 33 AP) on COCO. Despite their strong zero-shot performance, such foundation models may still be sub-optimal. For example, trucks on the web may be defined differently from trucks for a target application such as autonomous vehicle perception. We argue that the task of few-shot recognition can be reformulated as aligning foundation models to target concepts using a few examples. Interestingly, such examples can be multi-modal, using both text and visual cues, mimicking instructions that are often given to human annotators when defining a target concept of interest. Concretely, we propose Foundational FSOD, a new benchmark protocol that evaluates detectors pre-trained on any external data and fine-tuned on multi-modal (text and visual) K-shot examples per target class. We repurpose nuImages for Foundational FSOD, benchmark several popular open-source VLMs, and provide an empirical analysis of state-of-the-art methods. Lastly, we discuss our recent CVPR 2024 Foundational FSOD competition and share insights from the community. Notably, the winning team significantly outperforms our baseline by 23.3 mAP! Our code and dataset splits are available at https://github.com/anishmadan23/foundational_fsod

Motivation & Objective

• To address the misalignment between vision-language models (VLMs) and real-world target concepts, such as 'trailer' in autonomous driving vs. web-scale data.
• To challenge the conventional few-shot object detection (FSOD) paradigm that splits datasets into base and novel classes, arguing it does not reflect real-world deployment.
• To propose a new benchmark protocol—Foundational FSOD—where VLMs are fine-tuned directly on K-shot visual and language examples of target classes.
• To demonstrate that federated losses, leveraging the fact that FSOD datasets are inherently federated (each subset exhaustively annotated per class), significantly improve detection performance.

Proposed method

• Proposes Foundational FSOD, a new benchmark protocol that fine-tunes foundation VLMs directly on K-shot images and associated class names/descriptions, bypassing base/novel splits.
• Leverages the insight that existing FSOD benchmarks are federated datasets—each image subset is exhaustively annotated for a single category—enabling the use of federated loss functions.
• Applies federated loss functions (e.g., FedLoss, inverse FedLoss, pseudo-negatives) to improve model alignment with target concepts during fine-tuning.
• Uses vision-language models like Detic and GroundingDINO, initializing classifier heads with CLIP embeddings to enable zero-shot transfer and efficient few-shot adaptation.
• Employs ablation studies to identify optimal fine-tuning strategies, such as freezing the backbone and RPN while fine-tuning only the classifier projection and box regressor.
• Validates the approach on LVIS and nuImages, comparing performance to SOTA methods and oracle upper bounds with exhaustive annotations.

Figure 2 : Poor Alignment Between VLM and Target Concepts . Although VLMs show impressive zero-shot performance, they struggle when the target class is different from concepts encountered in web-scale training. On the left , we see that the nuImages dataset defines the cab of the truck as a separate

Experimental results

Research questions

• RQ1 Can zero-shot inference from vision-language models outperform state-of-the-art few-shot object detection methods without any fine-tuning?
• RQ2 Why do vision-language models still fail to detect target concepts accurately in domain-specific applications, despite strong zero-shot performance?
• RQ3 Can fine-tuning with K-shot multi-modal examples (images + text descriptions) improve concept alignment and detection performance in domain-specific settings?
• RQ4 How can the federated nature of existing few-shot object detection datasets be exploited to improve fine-tuning with minimal annotation cost?
• RQ5 To what extent can federated loss functions close the gap between current few-shot detection performance and the oracle upper bound with full annotations?

Key findings

• Zero-shot inference from pre-trained vision-language models like GroundingDINO achieves 48.3 AP on COCO, significantly outperforming the prior state-of-the-art method (33.1 AP).
• On the LVIS and nuImages benchmarks, the proposed method improves detection performance by 5.9 AP over prior work through K-shot fine-tuning and federated losses.
• Fine-tuning with CLIP-initialized classifier heads and only training the projection layer yields the best performance, highlighting the benefit of pre-trained language embeddings in few-shot adaptation.
• The use of federated losses—especially pseudo-negatives and inverse FedLoss—leads to substantial gains, with the best configuration reaching 16.67 AP on nuImages at 10 shots.
• The ablation study shows that freezing the backbone and RPN while fine-tuning only the classifier projection and box regressor yields optimal performance, minimizing overfitting on few-shot data.
• Even with strong zero-shot performance, rare categories (e.g., stroller) show lower detection accuracy, indicating a need for model ensembling or richer textual descriptions in future work.

Figure 3 : Identifying Negative Pseudo-Labels . On the left , we visualize the standard $K$ -shot detection setup, which we argue is actually a federated dataset [ 11 ] where one is given multiple mini-datasets of $K$ images. In this case, we visualize two $K=1$ datasets of bus and motorcycle . Impo