[Paper Review] Panorama to panorama matching for location recognition

Ahmet İşcen, Giorgos Tolias+3 more|arXiv (Cornell University)|Apr 21, 2017

Advanced Image and Video Retrieval Techniques24 references4 citations

TL;DR

This paper proposes a novel panorama-to-panorama matching method for location recognition by aggregating features from multiple panoramic views on both query and database sides using sum- and pseudo-inverse vectors. It achieves near-perfect recall (98% recall@1) on the Pittsburgh dataset using only four query images, significantly outperforming prior methods while reducing memory and computation costs through efficient feature aggregation without 3D reconstruction or supervision.

ABSTRACT

Location recognition is commonly treated as visual instance retrieval on "street view" imagery. The dataset items and queries are panoramic views, i.e. groups of images taken at a single location. This work introduces a novel panorama-to-panorama matching process, either by aggregating features of individual images in a group or by explicitly constructing a larger panorama. In either case, multiple views are used as queries. We reach near perfect location recognition on a standard benchmark with only four query views.

Motivation & Objective

• To improve location recognition performance in panoramic street-view imagery by leveraging multiple views from the same location.
• To eliminate the need for costly 3D reconstruction or supervised training by using natural, location-based grouping of images.
• To enhance retrieval efficiency and robustness by aggregating features on both query and database sides.
• To evaluate the effectiveness of implicit vs. explicit panorama construction in feature space for visual instance retrieval.

Proposed method

• Aggregates individual image features from a panoramic group into a single joint representation using sum-vector or pseudo-inverse (pinv-vector) methods.
• Applies the NetVLAD CNN descriptor to extract global features from each image before aggregation.
• Uses the Moore-Penrose pseudo-inverse to construct a theoretically optimized pinv-vector that performs well even with dissimilar or random vectors.
• Employs both implicit (feature-space) and explicit (image-stitching) panorama construction for comparison.
• Performs retrieval by computing cross-similarity between query and database joint representations using dot product or weighted similarity matrices.
• Evaluates performance under sparse query conditions by sampling only a subset of images from a location.

Experimental results

Research questions

• RQ1 Can joint representation of multiple panoramic views on both query and database sides significantly improve location recognition accuracy?
• RQ2 Does feature aggregation in feature space (without image stitching) outperform explicit panorama construction in retrieval performance?
• RQ3 How effective is the method when only a small number of query images (e.g., four) are available?
• RQ4 Can unsupervised, location-based grouping of images achieve state-of-the-art performance without 3D reconstruction or fine-tuning?
• RQ5 How does the performance of pinv-vector aggregation compare to NetVLAD-based descriptor extraction from a full panorama?

Key findings

• The proposed pan2pan method achieves 98% recall@1 on the Pittsburgh dataset using only four query images, demonstrating near-perfect location recognition.
• The method reduces memory usage by 24× and speeds up search by 24²× compared to baseline image-by-image retrieval.
• The pinv-vector aggregation method outperforms the sum-vector and NetVLAD-based approaches, especially in sparse query scenarios with only two or four images.
• Even with only four randomly sampled images from a location, the method achieves 99% recall@5, showing robustness to incomplete panoramas.
• The method significantly outperforms diffusion-based query expansion (96.5% vs. 91.9% recall@1), indicating that joint representation is more effective than iterative query refinement in this context.
• The results show that aggregating individual views into a pinv-vector can be more effective than extracting a single NetVLAD descriptor from a stitched panorama.