[Paper Review] Unifying Deep Local and Global Features for Image Search

Bingyi Cao, André Araujo+1 more|arXiv (Cornell University)|Jan 14, 2020

Advanced Image and Video Retrieval TechniquesComputer Science64 references34 citations

TL;DR

DELG unifies deep local and global image features into a single end-to-end trainable model, achieving state-of-the-art image retrieval and instance-level recognition by combining GeM pooling for global features and attentive local feature selection with an autoencoder-based dimensionality reduction.

ABSTRACT

Image retrieval is the problem of searching an image database for items that are similar to a query image. To address this task, two main types of image representations have been studied: global and local image features. In this work, our key contribution is to unify global and local features into a single deep model, enabling accurate retrieval with efficient feature extraction. We refer to the new model as DELG, standing for DEep Local and Global features. We leverage lessons from recent feature learning work and propose a model that combines generalized mean pooling for global features and attentive selection for local features. The entire network can be learned end-to-end by carefully balancing the gradient flow between two heads -- requiring only image-level labels. We also introduce an autoencoder-based dimensionality reduction technique for local features, which is integrated into the model, improving training efficiency and matching performance. Comprehensive experiments show that our model achieves state-of-the-art image retrieval on the Revisited Oxford and Paris datasets, and state-of-the-art single-model instance-level recognition on the Google Landmarks dataset v2. Code and models are available at https://github.com/tensorflow/models/tree/master/research/delf .

Motivation & Objective

• Motivate the integration of global and local image representations for retrieval efficiency and accuracy.
• Develop a unified CNN-based model that jointly learns global descriptors, keypoint attention, and local descriptors.
• Eliminate the need for patch-level supervision by carefully controlling gradient flow between global and local heads.
• Introduce a convolutional autoencoder to reduce local feature dimensionality without post-processing PCA.
• Demonstrate state-of-the-art performance on revisited Oxford/Paris and Google Landmarks v2 datasets.

Proposed method

• Use a CNN backbone to produce shallow (S) and deep (D) feature maps from which global and local features are derived.
• Global features are formed via generalized mean pooling (GeM) over D, followed by a learnable whitening layer to produce a 2048-dim global descriptor.
• Local features are obtained from S, passed through a 1x1 convolutional autoencoder to yield compact descriptors, with an attention map M selecting discriminative regions.
• Train end-to-end with three losses: ArcFace-based cosine classifier for global features, an autoencoder reconstruction loss for local features, and an attention-based softmax loss to encourage discriminative local selections.
• To maintain meaningful local representations during joint training, gradients from the local-attention and reconstruction losses are stopped from backpropagating into the CNN backbone.
• The model is trained with only image-level supervision, balancing gradient flow between global and local heads to avoid degrading the hierarchical feature representations.
• An optional binarization variant (DELG ⋆) stores local features in a binarized form for large-scale retrieval, with performance trade-offs discussed.

Experimental results

Research questions

• RQ1 Can a single end-to-end model learn both global descriptors and local, attention-weighted features for image retrieval effectively?
• RQ2 Does integrating an autoencoder for local descriptors and using gradient control allow joint optimization without patch-level supervision?
• RQ3 How do GeM pooling and ArcFace loss interact in a unified DELG model for robust global features?
• RQ4 What is the impact of end-to-end training on retrieval and recognition performance on standard benchmarks (Oxford/Paris revisited, GLDv2) compared to specialized, multi-model pipelines?

Key findings

• DELG achieves state-of-the-art results on Revisited Oxford, Revisited Paris, and Google Landmarks v2 with a single unified model.
• GeM pooling paired with ArcFace loss improves global feature performance, and an attention-guided local feature path with a lightweight autoencoder yields strong local descriptors.
• Joint training with proper gradient stopping preserves the hierarchical feature representations and yields competitive or superior performance compared to separately trained baselines.
• The unified model outperforms prior methods in both global-only and global-plus-local reranking settings, including large-scale scenarios with 1M distractors.
• A binarized variant (DELG ⋆) offers a memory-efficient option with competitive retrieval accuracy for very large databases.
• Local feature re-ranking using DELG notably boosts performance, especially in large-scale datasets.
• On GLDv2, DELG variants achieve leading mAP and μAP performance, with global-only and combined configurations showing strong results across retrieval and recognition tasks.