[Paper Review] Leveraging Deep Visual Descriptors for Hierarchical Efficient Localization

Paul-Edouard Sarlin, Frédéric Debraine+3 more|arXiv (Cornell University)|Sep 4, 2018

Robotics and Sensor-Based LocalizationEngineering44 citations

TL;DR

The paper presents a hierarchical visual localization pipeline that first uses learned global image descriptors to retrieve candidate places and then performs expensive local 2D-3D matching (SIFT) to estimate a centimeter-precise 6-DoF pose in city-scale, GPS-denied environments, while running in real-time on a mobile device.

ABSTRACT

Many robotics applications require precise pose estimates despite operating in large and changing environments. This can be addressed by visual localization, using a pre-computed 3D model of the surroundings. The pose estimation then amounts to finding correspondences between 2D keypoints in a query image and 3D points in the model using local descriptors. However, computational power is often limited on robotic platforms, making this task challenging in large-scale environments. Binary feature descriptors significantly speed up this 2D-3D matching, and have become popular in the robotics community, but also strongly impair the robustness to perceptual aliasing and changes in viewpoint, illumination and scene structure. In this work, we propose to leverage recent advances in deep learning to perform an efficient hierarchical localization. We first localize at the map level using learned image-wide global descriptors, and subsequently estimate a precise pose from 2D-3D matches computed in the candidate places only. This restricts the local search and thus allows to efficiently exploit powerful non-binary descriptors usually dismissed on resource-constrained devices. Our approach results in state-of-the-art localization performance while running in real-time on a popular mobile platform, enabling new prospects for robotics research.

Motivation & Objective

• Address the need for accurate 6-DoF localization in large, changing environments under limited mobile compute.
• Combine global learned descriptors for coarse place retrieval with expensive local features for precise pose estimation.
• Enable real-time operation on mobile hardware without requiring target-environment training.

Proposed method

• Compress a large image retrieval model into a mobile-friendly network via knowledge distillation to produce MobileNetVLAD global descriptors.
• Perform a covisibility-based clustering of retrieved priors to define candidate places.
• Use high-dimensional local descriptors (SIFT or FREAK) only within a reduced candidate set to compute 2D-3D matches.
• Solve a PnP-RANSAC pose estimation from consistent 2D-3D correspondences.
• Evaluate against baselines to show improved recall/precision and centimeter-accurate pose estimates.

Experimental results

Research questions

• RQ1 Can learned global descriptors enable tractable, coarse localization that reduces the search space for 2D-3D pose estimation?
• RQ2 Does a hierarchical, coarse-to-fine localization pipeline outperform existing resource-constrained methods in terms of accuracy and run-time on mobile hardware?
• RQ3 How do global descriptor distillation, covisibility clustering, and local descriptor choice (FREAK vs SIFT) affect localization performance?

Key findings

• MobileNetVLAD achieves retrieval recall close to NetVLAD but runs approximately 38x faster, enabling real-time mobile deployment.
• In full localization, MobileNetVLAD with SIFT (MNV+SIFT) achieves Recall@0.1m of 38.5%, Precision@0.1m of 80.5%, and Median error of 0.029 m, outperforming Direct+FREAK and approaching the perfect-SIFT upper bound.
• Using 5–10 prior frames provides strong pose estimates while keeping run-time low; increasing the number of priors yields diminishing returns and higher latency.
• The full pipeline runs at 2.2 FPS on an NVIDIA Jetson TX2, with local matching as the bottleneck and potential for speedups at the cost of some accuracy.
• Compared to Maplab’s Direct+FREAK baseline, the proposed method improves recall by a substantial margin (18.4% in the reported results).
• Even with real-time constraints, the system delivers centimeter-precise poses in city-scale GPS-denied environments without requiring target-environment training.