[Paper Review] Query-Adaptive R-CNN for Open-Vocabulary Object Detection and Retrieval.

Ryota Hinami, Shin’ichi Satoh|arXiv (Cornell University)|Nov 27, 2017

Multimodal Machine Learning ApplicationsComputer Science31 references3 citations

TL;DR

This paper proposes Query-Adaptive R-CNN, a novel end-to-end framework that extends Faster R-CNN to open-vocabulary object detection by generating detector weights from textual queries. It achieves state-of-the-art performance on Flickr30k Entities and retrieves and localizes objects from one million images in 0.5 seconds using query-adaptive classification and negative phrase augmentation for improved discrimination.

ABSTRACT

We address the problem of open-vocabulary object retrieval and localization, which is to retrieve and localize objects from a very large-scale image database immediately by a textual query (e.g., a word or phrase). We first propose Query-Adaptive R-CNN, a simple yet strong framework for open-vocabulary object detection. Query-Adaptive R-CNN is a simple extension of Faster R-CNN from closed-vocabulary to open-vocabulary object detection: instead of learning a class-specific classifier and regressor, we learn a detector generator that transforms a text into classifier and regressor weights. All of its components can be learned in an end-to-end manner. Even with its simple architecture, it outperforms all state-of-the-art methods in the Flickr30k Entities phrase localization task. In addition, we propose negative phrase augmentation, a generic approach for exploiting hard negatives in the training of open-vocabulary object detection that significantly improves the discriminative ability of the generated classifier. We show that our system can retrieve and localize objects specified by a textual query from one million images in only 0.5 seconds.

Motivation & Objective

• To enable open-vocabulary object detection and retrieval from large-scale image databases using natural language queries.
• To address the challenge of generalizing object detection beyond fixed, closed-set classes by enabling dynamic adaptation to unseen textual queries.
• To improve model discriminative ability in open-vocabulary settings through effective hard negative mining during training.
• To achieve fast, end-to-end inference for real-time retrieval and localization across massive image collections.

Proposed method

• Query-Adaptive R-CNN replaces class-specific classifiers and regressors in Faster R-CNN with a detector generator that produces classifier and regressor weights from textual queries.
• The detector generator is trained end-to-end, enabling joint optimization of visual and textual representations for query-specific detection.
• Negative phrase augmentation is introduced as a data-augmentation strategy to explicitly model hard negative phrases during training, enhancing classifier generalization.
• The framework leverages pre-trained vision and text encoders to embed images and queries into a shared embedding space for alignment.
• The system uses a region proposal network (RPN) to generate candidate object proposals, which are then scored using query-generated weights.
• Inference is highly efficient, enabling retrieval and localization from one million images in under 0.5 seconds.

Experimental results

Research questions

• RQ1 Can a simple, end-to-end framework adapt object detection to open-vocabulary queries without requiring retraining for each new class?
• RQ2 How effectively can a detector generator produce accurate classifier and regressor weights from natural language queries?
• RQ3 To what extent does negative phrase augmentation improve discrimination in open-vocabulary detection?
• RQ4 What is the inference speed and scalability of the system on large-scale image databases?

Key findings

• Query-Adaptive R-CNN achieves state-of-the-art performance on the Flickr30k Entities phrase localization benchmark, outperforming all prior methods.
• The model retrieves and localizes objects from one million images in just 0.5 seconds, demonstrating high inference efficiency.
• Negative phrase augmentation significantly improves the discriminative capability of the classifier, especially for ambiguous or rare phrases.
• The end-to-end trainable detector generator enables strong generalization to unseen queries without fine-tuning.
• The framework maintains high accuracy even under challenging open-vocabulary conditions with diverse and complex textual queries.