[Paper Review] Open Vocabulary Object Detection with Pseudo Bounding-Box Labels

Mingfei Gao, Xing Chen+5 more|arXiv (Cornell University)|Nov 18, 2021

Multimodal Machine Learning Applications5 citations

TL;DR

This paper proposes a method to generate pseudo bounding-box labels for open vocabulary object detection using pre-trained vision-language models and large-scale image-caption pairs. By leveraging the localization capability of these models to automatically generate diverse, scalable training annotations, the approach improves detection performance on novel categories, outperforming state-of-the-art methods by 8% mAP on COCO novel categories without requiring human-annotated boxes for base classes.

ABSTRACT

Despite great progress in object detection, most existing methods work only on a limited set of object categories, due to the tremendous human effort needed for bounding-box annotations of training data. To alleviate the problem, recent open vocabulary and zero-shot detection methods attempt to detect novel object categories beyond those seen during training. They achieve this goal by training on a pre-defined base categories to induce generalization to novel objects. However, their potential is still constrained by the small set of base categories available for training. To enlarge the set of base classes, we propose a method to automatically generate pseudo bounding-box annotations of diverse objects from large-scale image-caption pairs. Our method leverages the localization ability of pre-trained vision-language models to generate pseudo bounding-box labels and then directly uses them for training object detectors. Experimental results show that our method outperforms the state-of-the-art open vocabulary detector by 8% AP on COCO novel categories, by 6.3% AP on PASCAL VOC, by 2.3% AP on Objects365 and by 2.8% AP on LVIS. Code is available at https://github.com/salesforce/PB-OVD.

Motivation & Objective

• To address the limitation of existing open vocabulary detection methods that rely on a small, human-annotated set of base categories.
• To reduce dependency on expensive human-annotated bounding boxes by automatically generating pseudo labels from large-scale image-caption datasets.
• To improve generalization to novel object categories by expanding the diversity and scale of base categories used during training.
• To evaluate the effectiveness of pseudo bounding-box labels in enhancing detection performance across multiple benchmark datasets.
• To investigate the impact of different text encoders and data scales on the quality and generalization of pseudo labels.

Proposed method

• Leverage a pre-trained vision-language model to generate activation maps (via Grad-CAM) corresponding to objects mentioned in image captions.
• Convert the activation maps into pseudo bounding-box labels for the corresponding object categories in the image-caption pairs.
• Use the automatically generated pseudo labels to train an open vocabulary object detector, enabling it to generalize to novel categories.
• Fine-tune the detector with human-annotated boxes when available, but the method remains effective even without such fine-tuning.
• Scale the pseudo label generation process to large-scale image-caption datasets (e.g., COCO Captions) to increase the diversity and number of base categories.
• Evaluate the impact of different text encoders (e.g., CLIP vs. BERT) and data volume on the performance of the final detector.

Experimental results

Research questions

• RQ1 Can pseudo bounding-box labels generated from image-caption pairs improve open vocabulary object detection performance?
• RQ2 How does the scale and diversity of pseudo-labeled base categories affect detection generalization to novel categories?
• RQ3 Does using a pre-trained vision-language model’s localization ability lead to more accurate and generalizable pseudo labels than human-annotated baselines?
• RQ4 How does the choice of text encoder (e.g., CLIP vs. BERT) influence the quality of generated pseudo labels and downstream detection performance?
• RQ5 Can the method outperform state-of-the-art open vocabulary detectors even without fine-tuning on human-annotated base categories?

Key findings

• The proposed method outperforms the state-of-the-art open vocabulary detector by 8% mAP on COCO novel categories when both are fine-tuned with COCO base categories.
• Even without fine-tuning on COCO base categories, the method still achieves 3% higher mAP than the fine-tuned SOTA baseline on COCO novel categories.
• Using a larger vocabulary of object categories during pre-training (beyond COCO) improves performance by 2.3% on Objects365 and 2.7% on LVIS.
• Increasing the amount of image-caption data used for pseudo label generation improves detection performance by approximately 2% on the target set.
• The method with CLIP as the text encoder outperforms the one using BERT by a small margin, indicating that the performance gain is primarily due to the scale and diversity of pseudo labels rather than the encoder choice.
• Qualitative results show that the pseudo label generator successfully localizes objects not in COCO’s original category list (e.g., slippers, pot, pie), demonstrating its ability to generalize to out-of-vocabulary objects.