[Paper Review] Levelling the Playing Field: A Comprehensive Comparison of Visual Place Recognition Approaches under Changing Conditions

Mubariz Zaffar, Ahmad Khaliq+3 more|arXiv (Cornell University)|Mar 21, 2019

Advanced Image and Video Retrieval Techniques48 references35 citations

TL;DR

The paper benchmarked 10 state-of-the-art Visual Place Recognition (VPR) techniques on three challenging datasets using standardized metrics: matching performance (AUC), matching time, and memory footprint, to provide a unified comparison and guide future VPR research.

ABSTRACT

In recent years there has been significant improvement in the capability of Visual Place Recognition (VPR) methods, building on the success of both hand-crafted and learnt visual features, temporal filtering and usage of semantic scene information. The wide range of approaches and the relatively recent growth in interest in the field has meant that a wide range of datasets and assessment methodologies have been proposed, often with a focus only on precision-recall type metrics, making comparison difficult. In this paper we present a comprehensive approach to evaluating the performance of 10 state-of-the-art recently-developed VPR techniques, which utilizes three standardized metrics: (a) Matching Performance b) Matching Time c) Memory Footprint. Together this analysis provides an up-to-date and widely encompassing snapshot of the various strengths and weaknesses of contemporary approaches to the VPR problem. The aim of this work is to help move this particular research field towards a more mature and unified approach to the problem, enabling better comparison and hence more progress to be made in future research.

Motivation & Objective

• Assess the performance of 10 recent VPR techniques under challenging appearance changes.
• Provide a fair, unified evaluation platform by keeping datasets, ground truth, and metrics constant.
• Quantify trade-offs between accuracy, speed, and memory usage for deployment considerations.

Proposed method

• Evaluate 10 VPR techniques on three public datasets under standardized conditions: Berlin Kudamm, Gardens Point, and Nordland.
• Use three metrics: Matching Performance (AUC of precision-recall), Matching Time (query encoding and descriptor matching), and Memory Footprint (descriptor size).
• Implement and configure each technique as described: HOG, Seq-SLAM, AlexNet, NetVLAD, AMOSNet, HybridNet, Cross-Region-BOW, R-MAC, Region-VLAD, CALC.
• Compare techniques on grounds of sensitivity to viewpoint/season/illumination changes and presence of dynamic objects.

Experimental results

Research questions

• RQ1 How do top VPR techniques compare in terms of accuracy under extreme appearance variations?
• RQ2 What are the computational trade-offs (time and memory) across methods when deployed on a uniform platform?
• RQ3 Which methods offer the best balance between matching performance and resource requirements for real-time deployment?

Key findings

• NetVLAD achieves state-of-the-art performance on the Berlin Kudamm dataset.
• Region-VLAD and Cross-Region-BoW closely follow NetVLAD on Berlin Kudamm, with AMOSNet and HybridNet offering competitive results but degraded by extreme viewpoint variation.
• Most methods perform well on Gardens Point despite illumination changes, with Cross-Region-BoW achieving strong results.
• Region-VLAD provides top performance on Nordland, with Net-VLAD and Cross-Region-BOW also competitive.
• CNN-based methods incur higher encoding time and larger memory footprints than handcrafted methods, while CALC shows notably faster encoding time.
• Cross-Region-BOW and Region-VLAD have large descriptor sizes, impacting deployment on resource-constrained platforms.