[Paper Review] Real-time Image-based 6-DOF Localization in Large-Scale Environments

Hyon Lim, Sudipta N. Sinha+2 more|arXiv (Cornell University)|Mar 20, 2012

Robotics and Sensor-Based Localization18 citations

TL;DR

This paper presents a real-time, monocular image-based 6-DOF localization method for large-scale environments using an SfM-reconstructed point cloud. By interleaving a fast binary descriptor tracker with sparse, offline-indexed DAISY descriptor matching across a frame window, it achieves stable, low-latency localization at over 30 Hz on a laptop and 12 Hz on a mobile platform, eliminating the need for online scale-invariant feature extraction.

ABSTRACT

We present a real-time approach for image-based localization within large scenes that have been reconstructed offline using structure from motion (Sfm). From monocular video, our method continuously computes a precise 6-DOF camera pose, by efficiently tracking natural features and matching them to 3D points in the Sfm point cloud. Our main contribution lies in efficiently interleaving a fast keypoint tracker that uses inexpensive binary feature descriptors with a new approach for direct 2D-to-3D matching. The 2D-to-3D matching avoids the need for online extraction of scale-invariant features. Instead, offline we construct an indexed database containing multiple DAISY descriptors per 3D point extracted at multiple scales. The key to the efficiency of our method lies in invoking DAISY descriptor extraction and matching sparingly during localization, and in distributing this computation over a window of successive frames. This enables the algorithm to run in real-time, without fluctuations in the latency over long durations. We evaluate the method in large indoor and outdoor scenes. Our algorithm runs at over 30 Hz on a laptop and at 12 Hz on a low-power, mobile computer suitable for onboard computation on a quadrotor micro aerial vehicle.

Motivation & Objective

• To enable real-time, accurate 6-DOF camera pose estimation in large-scale indoor and outdoor environments from monocular video.
• To overcome the computational bottleneck of online scale-invariant feature extraction during localization.
• To maintain consistent low latency over long durations without performance fluctuations.
• To support deployment on low-power mobile platforms, such as quadrotor UAVs, for onboard navigation.

Proposed method

• The method uses a fast keypoint tracker with inexpensive binary feature descriptors for real-time 2D feature tracking across video frames.
• It performs 2D-to-3D matching by querying an offline-indexed database of multiple DAISY descriptors per 3D point, extracted at multiple scales.
• DAISY descriptor extraction and matching are invoked sparingly and distributed across a sliding window of successive frames to balance accuracy and efficiency.
• The system avoids online feature extraction by precomputing and indexing descriptors, reducing runtime computation.
• The algorithm maintains real-time performance by offloading descriptor computation and balancing it over time.
• Pose estimation is continuously refined using tracked 2D features and their 3D correspondences from the SfM point cloud.

Experimental results

Research questions

• RQ1 Can real-time 6-DOF localization be achieved in large-scale environments using only monocular video and an offline SfM reconstruction?
• RQ2 How can 2D-to-3D matching be made efficient enough to run in real time without online scale-invariant feature extraction?
• RQ3 What is the performance trade-off between accuracy and computational load when distributing descriptor computation over a frame window?
• RQ4 Can the method maintain consistent latency over long durations in dynamic environments?
• RQ5 Is the approach suitable for deployment on low-power mobile platforms like micro aerial vehicles?

Key findings

• The method achieves over 30 Hz localization performance on a standard laptop, enabling real-time operation.
• It runs at 12 Hz on a low-power mobile computer, making it viable for onboard computation on a quadrotor micro aerial vehicle.
• The system maintains stable latency without fluctuations, even during prolonged operation.
• By precomputing and indexing DAISY descriptors at multiple scales, the method avoids costly online feature extraction.
• The approach enables accurate 6-DOF localization in both large indoor and outdoor scenes using only monocular video and SfM data.
• The use of a frame window for distributing descriptor computation ensures high efficiency while preserving tracking robustness.