[Paper Review] Communication constrained cloud-based long-term visual localization in real time

Xiaqing Ding, Yue Wang+3 more|arXiv (Cornell University)|Mar 10, 2019

Robotics and Sensor-Based Localization24 references4 citations

TL;DR

This paper proposes a cloud-based visual localization system that enables real-time, drift-free localization for mobile robots under communication constraints. By using a dual sliding-window architecture—onboard for local pose estimation and on the cloud for aggregated virtual submap alignment—the system achieves robust long-term localization even with high latency, packet loss, and limited bandwidth, validated on seasonal and challenging network conditions.

ABSTRACT

Visual localization is one of the primary capabilities for mobile robots. Long-term visual localization in real time is particularly challenging, in which the robot is required to efficiently localize itself using visual data where appearance may change significantly over time. In this paper, we propose a cloud-based visual localization system targeting at long-term localization in real time. On the robot, we employ two estimators to achieve accurate and real-time performance. One is a sliding-window based visual inertial odometry, which integrates constraints from consecutive observations and self-motion measurements, as well as the constraints induced by localization on the cloud. This estimator builds a local visual submap as the virtual observation which is then sent to the cloud as new localization constraints. The other one is a delayed state Extended Kalman Filter to fuse the pose of the robot localized from the cloud, the local odometry and the high-frequency inertial measurements. On the cloud, we propose a longer sliding-window based localization method to aggregate the virtual observations for larger field of view, leading to more robust alignment between virtual observations and the map. Under this architecture, the robot can achieve drift-free and real-time localization using onboard resources even in a network with limited bandwidth, high latency and existence of package loss, which enables the autonomous navigation in real-world environment. We evaluate the effectiveness of our system on a dataset with challenging seasonal and illuminative variations. We further validate the robustness of the system under challenging network conditions.

Motivation & Objective

• Address the challenge of long-term visual localization in real time, where appearance variations (e.g., lighting, season) degrade performance.
• Overcome the limitations of onboard processing in resource-constrained robots by offloading computation to the cloud.
• Ensure robustness under real-world network conditions such as high latency, packet loss, and limited bandwidth.
• Achieve drift-free localization by fusing cloud-based global pose estimates with local visual-inertial odometry.
• Design a communication-efficient system that minimizes data transmission while maintaining localization accuracy.

Proposed method

• Deploy a sliding-window-based visual-inertial odometry (VIO) on the robot to estimate local pose using consecutive frames and inertial measurements.
• Use a delayed state Extended Kalman Filter (EKF) on the robot to fuse cloud-provided global poses, local VIO, and high-frequency IMU data.
• Construct virtual observations (local visual submaps) on the robot and transmit them to the cloud as compressed, sparsified data to reduce bandwidth usage.
• Implement a longer sliding-window localization method on the cloud to aggregate virtual observations, enabling larger field-of-view alignment with the map.
• Apply an information sparsification technique to compress and reduce transmitted data, ensuring compatibility with low-bandwidth networks.
• Integrate network-aware compensation in the EKF to handle latency and packet loss, maintaining estimation stability.

Experimental results

Research questions

• RQ1 How can long-term visual localization be made robust to significant appearance changes (e.g., seasonal and illuminative variations) in real time?
• RQ2 What system architecture enables drift-free localization on resource-limited robots under communication constraints?
• RQ3 How can data transmission be minimized without sacrificing localization accuracy in cloud-based visual localization?
• RQ4 To what extent can the system tolerate network latency and packet loss in real-world deployments?
• RQ5 What is the optimal balance between sliding window length on the cloud and localization accuracy and robustness?

Key findings

• The system achieves drift-free, real-time localization on the robot even under 50% packet loss and up to 3 seconds of network latency.
• Localization performance remains stable and accurate under 70% packet loss, demonstrating strong resilience to data loss.
• With network latency up to 5 seconds, the system still provides rough but usable localization results, exceeding typical real-world network conditions.
• The cloud-side sliding window length of 40 provides the best trade-off between robustness and accuracy, outperforming shorter (10–20) and longer (50) windows.
• The dual sliding-window architecture significantly improves alignment robustness by aggregating multiple virtual observations into a larger field of view.
• The information sparsification method effectively reduces data transmission while maintaining localization accuracy, satisfying low-bandwidth constraints.