[Paper Review] Open-Vocabulary Object Detection Using Captions

Alireza Zareian, Kevin Dela Rosa+2 more|arXiv (Cornell University)|Nov 20, 2020

Multimodal Machine Learning Applications41 references17 citations

TL;DR

This paper proposes Open-Vocabulary Object Detection (OVD), a novel two-stage method that leverages image-caption pairs to pre-train a visual-semantic space and fine-tunes it on bounding box annotations for base classes. The approach achieves 27.5% mAP on zero-shot detection and 39.9% mAP in generalized zero-shot settings, significantly outperforming prior state-of-the-art methods.

ABSTRACT

Despite the remarkable accuracy of deep neural networks in object detection, they are costly to train and scale due to supervision requirements. Particularly, learning more object categories typically requires proportionally more bounding box annotations. Weakly supervised and zero-shot learning techniques have been explored to scale object detectors to more categories with less supervision, but they have not been as successful and widely adopted as supervised models. In this paper, we put forth a novel formulation of the object detection problem, namely open-vocabulary object detection, which is more general, more practical, and more effective than weakly supervised and zero-shot approaches. We propose a new method to train object detectors using bounding box annotations for a limited set of object categories, as well as image-caption pairs that cover a larger variety of objects at a significantly lower cost. We show that the proposed method can detect and localize objects for which no bounding box annotation is provided during training, at a significantly higher accuracy than zero-shot approaches. Meanwhile, objects with bounding box annotation can be detected almost as accurately as supervised methods, which is significantly better than weakly supervised baselines. Accordingly, we establish a new state of the art for scalable object detection.

Motivation & Objective

• To address the high cost and limited scalability of supervised object detection, which requires extensive bounding box annotations for each new category.
• To enable detection of objects beyond the base classes used during training, without requiring additional annotated data.
• To unify recognition and localization by leveraging weakly supervised image-caption data for open-vocabulary semantic learning and supervised bounding box data for accurate localization.
• To establish a new paradigm that generalizes beyond zero-shot and weakly supervised detection by combining both data sources effectively.

Proposed method

• Pre-train a Faster R-CNN backbone on a large-scale image-caption dataset to learn a rich, open-vocabulary visual-semantic embedding space.
• Fine-tune the model on a smaller dataset with bounding box annotations for base classes, while preserving the pre-trained visual-semantic knowledge.
• Introduce a Visual-to-Language (V2L) projection layer that maps image features to class embeddings, enabling zero-shot generalization to unseen categories.
• Freeze the V2L layer during fine-tuning to maintain generalization capacity to target classes, preventing overfitting to base classes.
• Use a multimodal transformer module to jointly optimize grounding and auxiliary objectives during pretraining, enhancing semantic alignment.
• Transfer both ResNet and V2L weights from pretraining to downstream detection, ensuring effective knowledge transfer.

Experimental results

Research questions

• RQ1 Can a model trained on image-caption pairs generalize to detect and localize object categories not seen during training, without any bounding box annotations for those categories?
• RQ2 How does pretraining on image-caption data improve zero-shot detection performance compared to standard zero-shot learning methods?
• RQ3 To what extent can weakly supervised signals from captions improve detection performance when combined with limited bounding box annotations?
• RQ4 What components of the pretraining and fine-tuning pipeline are most critical for achieving strong zero-shot generalization?

Key findings

• OVR-CNN achieves 27.5% mAP on zero-shot detection, a 17 percentage point improvement over the previous state of the art (10% mAP).
• In generalized zero-shot settings, OVR-CNN reaches 39.9% mAP, significantly outperforming weakly supervised baselines (26% mAP).
• Ablation studies show that pretraining on image-caption data is essential, as removing it reduces mAP to 18.1% on all classes.
• The V2L layer is critical for generalization; unfreezing it during fine-tuning causes a drop in zero-shot performance from 27.5% to 23.4% mAP.
• Even with low-quality Conceptual Captions, OVR-CNN achieves 34.3% mAP, outperforming all ZSD baselines, demonstrating robustness to noisy data.
• Transferring only ResNet weights is insufficient; transferring the V2L layer is essential for effective knowledge transfer and open-vocabulary generalization.