[Paper Review] Detect-to-Retrieve: Efficient Regional Aggregation for Image Search

Marvin Teichmann, André Araujo+2 more|arXiv (Cornell University)|Dec 4, 2018

Advanced Image and Video Retrieval Techniques40 references4 citations

TL;DR

This paper proposes Detect-to-Retrieve, a method that uses a trained landmark detector to select relevant image regions for efficient and accurate image retrieval. By applying a novel regional aggregated selective match kernel (R-ASMK), it improves retrieval accuracy without increasing dimensionality, outperforming prior methods by 9.3% on Revisited Oxford-Hard and 1.9% on Revisited Paris-Hard, while using a new dataset of 86k manually annotated landmark bounding boxes.

ABSTRACT

Retrieving object instances among cluttered scenes efficiently requires compact yet comprehensive regional image representations. Intuitively, object semantics can help build the index that focuses on the most relevant regions. However, due to the lack of bounding-box datasets for objects of interest among retrieval benchmarks, most recent work on regional representations has focused on either uniform or class-agnostic region selection. In this paper, we first fill the void by providing a new dataset of landmark bounding boxes, based on the Google Landmarks dataset, that includes $86k$ images with manually curated boxes from $15k$ unique landmarks. Then, we demonstrate how a trained landmark detector, using our new dataset, can be leveraged to index image regions and improve retrieval accuracy while being much more efficient than existing regional methods. In addition, we introduce a novel regional aggregated selective match kernel (R-ASMK) to effectively combine information from detected regions into an improved holistic image representation. R-ASMK boosts image retrieval accuracy substantially with no dimensionality increase, while even outperforming systems that index image regions independently. Our complete image retrieval system improves upon the previous state-of-the-art by significant margins on the Revisited Oxford and Paris datasets. Code and data available at the project webpage: https://github.com/tensorflow/models/tree/master/research/delf.

Motivation & Objective

• To address the challenge of retrieving small or cluttered objects in images by improving region selection for retrieval.
• To reduce the inefficiency and high memory cost of existing regional search methods that use uniform or class-agnostic region selection.
• To develop a method that leverages object semantics to focus indexing on relevant regions, improving accuracy with minimal computational overhead.
• To introduce a new dataset of 86k manually annotated landmark bounding boxes to support training robust object detectors for retrieval.
• To design a regional aggregation mechanism that enhances holistic image representations using detected regions without increasing descriptor dimensionality.

Proposed method

• The method introduces a new dataset of 86k images with manually curated bounding boxes for 15,000 unique landmarks, enabling training of accurate landmark detectors.
• A trained object detector (Faster R-CNN and SSD) is used to identify salient regions in images, replacing fixed or class-agnostic region selection.
• The regional aggregated selective match kernel (R-ASMK) combines per-region VLAD descriptors via sum pooling and per-visual word normalization to produce a unified, discriminative image representation.
• R-ASMK reweights local features based on their spatial location within detected regions, enhancing relevance of features in salient areas.
• The system uses deep local features (DELF) and applies R-ASMK to improve similarity estimation during retrieval, without increasing descriptor size.
• The approach integrates region-based representation learning with selective match kernels, enabling efficient filtering and re-ranking in image retrieval pipelines.

Experimental results

Research questions

• RQ1 Can a trained object detector improve region selection for image retrieval in cluttered scenes compared to fixed or class-agnostic methods?
• RQ2 Can a region-aware representation improve retrieval accuracy without increasing descriptor dimensionality?
• RQ3 Does leveraging object semantics via a dedicated detection model lead to better performance on standard benchmarks like Revisited Oxford and Paris?
• RQ4 How does the proposed R-ASMK kernel compare to independent region indexing and global representation methods in terms of accuracy and efficiency?
• RQ5 To what extent does a human-annotated landmark bounding box dataset improve detector generalization and retrieval performance?

Key findings

• The proposed method improves mean average precision by 9.3% absolute on the Revisited Oxford-Hard dataset, significantly outperforming previous state-of-the-art.
• On the Revisited Paris-Hard dataset, the method achieves a 1.9% absolute improvement in mean average precision over prior methods.
• The R-ASMK kernel outperforms systems that index regions independently, even without increasing descriptor dimensionality.
• The trained landmark detector achieves around 85% mAP after 600k training steps, with SSD-based models showing faster convergence and slightly better performance than Faster R-CNN.
• Visualizations show that the detector focuses on prominent landmarks, while RMAC and Selective Search produce less relevant or poorly aligned regions.
• The method maintains efficiency, with only a modest increase in database size, while achieving substantial accuracy gains over existing regional search techniques.