[Paper Review] Stochastic Attraction-Repulsion Embedding for Large Scale Image Localization

Liu Liu, Hongdong Li+1 more|arXiv (Cornell University)|Aug 27, 2018

Advanced Image and Video Retrieval Techniques36 references4 citations

TL;DR

This paper proposes Stochastic Attraction and Repulsion Embedding (SARE), a novel loss function that enhances image representation learning for large-scale image-based localization by modeling intra-place similarity and inter-place dissimilarity as a probabilistic distribution over L2 distances. SARE minimizes KL divergence between predicted and ground-truth matchability distributions, achieving state-of-the-art performance, including 3rd place in the 2018 Google Landmark Retrieval Challenge.

ABSTRACT

This paper tackles the problem of large-scale image-based localization (IBL) where the spatial location of a query image is determined by finding out the most similar reference images in a large database. For solving this problem, a critical task is to learn discriminative image representation that captures informative information relevant for localization. We propose a novel representation learning method having higher location-discriminating power. It provides the following contributions: 1) we represent a place (location) as a set of exemplar images depicting the same landmarks and aim to maximize similarities among intra-place images while minimizing similarities among inter-place images; 2) we model a similarity measure as a probability distribution on L_2-metric distances between intra-place and inter-place image representations; 3) we propose a new Stochastic Attraction and Repulsion Embedding (SARE) loss function minimizing the KL divergence between the learned and the actual probability distributions; 4) we give theoretical comparisons between SARE, triplet ranking and contrastive losses. It provides insights into why SARE is better by analyzing gradients. Our SARE loss is easy to implement and pluggable to any CNN. Experiments show that our proposed method improves the localization performance on standard benchmarks by a large margin. Demonstrating the broad applicability of our method, we obtained the third place out of 209 teams in the 2018 Google Landmark Retrieval Challenge. Our code and model are available at https://github.com/Liumouliu/deepIBL.

Motivation & Objective

• To improve discriminative image representation learning for large-scale image-based localization (IBL) by explicitly modeling intra-place attraction and inter-place repulsion.
• To address the limitation of existing methods that rely on triplet ranking or contrastive losses without a principled probabilistic formulation.
• To develop a differentiable, plug-and-play loss function that enhances CNN-based feature embeddings for localization without architectural changes.
• To validate the superiority of SARE through extensive benchmarks and real-world challenges, including the Google Landmark Retrieval Challenge.
• To demonstrate generalization of SARE-trained models on standard image retrieval datasets without fine-tuning.

Proposed method

• The method models image similarity as a binary classification task: 1 for intra-place pairs (attraction), 0 for inter-place pairs (repulsion).
• It defines a probability distribution over L2 distances between feature embeddings of query-positive and query-negative image pairs.
• The SARE loss is formulated as the Kullback-Leibler (KL) divergence between the predicted probability distribution and a ground-truth matchability distribution.
• The loss function is designed to be compatible with any CNN architecture and can be easily integrated into existing IBL pipelines.
• The framework supports multiple negative samples per query, enabling competition among multiple distinct places.
• The method uses VGG16 with NetVLAD pooling and L2-normalized features, followed by L2 distance computation and probabilistic matching.

Experimental results

Research questions

• RQ1 How can we improve the discriminative power of image embeddings for large-scale image localization beyond traditional metric learning losses?
• RQ2 Can a probabilistic formulation of intra-place attraction and inter-place repulsion lead to better localization performance than standard contrastive or triplet losses?
• RQ3 Does the SARE loss generalize well to standard image retrieval benchmarks without fine-tuning?
• RQ4 What is the theoretical advantage of SARE over existing metric learning objectives like contrastive and triplet losses?
• RQ5 Can SARE achieve state-of-the-art performance in real-world challenges such as the Google Landmark Retrieval Challenge?

Key findings

• SARE achieved 3rd place out of 209 teams in the 2018 Google Landmark Retrieval Challenge, demonstrating strong real-world performance.
• On the Sf-0 benchmark, SARE achieved a mean average precision (mAP) of 80.71% on full images and 81.07% on cropped images, outperforming NetVLAD and CRN.
• On the Oxford5k and Paris6k datasets, SARE achieved mAP of 71.66% and 82.03% respectively, surpassing baseline methods like NetVLAD and CRN.
• The method showed superior generalization on image retrieval tasks, with mAP consistently higher than off-the-shelf NetVLAD and CRN on all three benchmarks.
• Qualitative analysis confirmed that SARE focuses on distinctive architectural details in images, while NetVLAD emphasizes local features.
• Theoretical analysis showed that SARE’s gradient dynamics are more favorable than those of contrastive and triplet losses, explaining its empirical superiority.