[Paper Review] Lvio-Fusion: A Self-adaptive Multi-sensor Fusion SLAM Framework Using Actor-critic Method

Yupeng Jia, Haiyong Luo+6 more|arXiv (Cornell University)|Jun 12, 2021

Robotic Path Planning Algorithms4 citations

TL;DR

Lvio-Fusion proposes a tightly coupled, self-adaptive multi-sensor SLAM framework fusing stereo camera, LiDAR, IMU, and GPS via factor graph optimization. It employs a segmented global pose graph optimization to correct drift in urban scenes and uses a reinforcement learning-based actor-critic agent to dynamically adjust sensor weights, achieving high accuracy and robustness on KITTI and KAIST datasets with real-time performance.

ABSTRACT

State estimation with sensors is essential for mobile robots. Due to different performance of sensors in different environments, how to fuse measurements of various sensors is a problem. In this paper, we propose a tightly coupled multi-sensor fusion framework, Lvio-Fusion, which fuses stereo camera, Lidar, IMU, and GPS based on the graph optimization. Especially for urban traffic scenes, we introduce a segmented global pose graph optimization with GPS and loop-closure, which can eliminate accumulated drifts. Additionally, we creatively use a actor-critic method in reinforcement learning to adaptively adjust sensors' weight. After training, actor-critic agent can provide the system better and dynamic sensors' weight. We evaluate the performance of our system on public datasets and compare it with other state-of-the-art methods, which shows that the proposed method achieves high estimation accuracy and robustness to various environments. And our implementations are open source and highly scalable.

Motivation & Objective

• To address the challenge of inconsistent sensor performance across diverse environments in multi-sensor SLAM.
• To reduce accumulated drift in large-scale urban environments using GPS and loop-closure constraints.
• To develop a dynamic, self-adaptive sensor fusion mechanism that adjusts weights based on environmental conditions.
• To enable real-time, high-precision trajectory estimation using tightly coupled fusion of stereo vision, LiDAR, IMU, and GPS.

Proposed method

• The system uses visual-inertial odometry to extract keyframes and performs factor graph optimization with stereo camera, LiDAR, IMU, and GPS as factors.
• A segmented global pose graph optimization is applied to different trajectory segments based on motion characteristics, improving drift correction in urban scenes.
• An actor-critic reinforcement learning agent is trained in parallel to dynamically adjust the weights of sensor factors based on real-time performance feedback.
• The agent is trained on KITTI sequences 02, 05, 06, 07, 09 and validated on sequence 00, using RPE as the evaluation metric during validation.
• The lightweight TD3-based agent processes visual and LiDAR features and IMU pre-integration to compute adaptive weights in 5.56ms per keyframe.
• The framework supports plug-and-play integration of new sensors by treating them as additional factors in the factor graph.

Experimental results

Research questions

• RQ1 How can multi-sensor SLAM systems achieve robust and accurate state estimation across diverse environments with varying sensor reliability?
• RQ2 Can a reinforcement learning-based agent effectively adapt sensor fusion weights in real time to improve SLAM accuracy?
• RQ3 How does segmented global optimization with GPS and loop closure reduce accumulated drift in urban traffic scenes?
• RQ4 What is the performance gain of using a self-adaptive sensor weighting mechanism compared to fixed-weight fusion in challenging urban environments?

Key findings

• On the KITTI dataset, the self-adaptive Lvio-Fusion reduced the RPE RMSE from 1.73m to 1.39m in sequence 00, demonstrating a 20% improvement in accuracy.
• In the KAIST urban dataset, Lvio-Fusion achieved an ATE RMSE of 6.31m, significantly outperforming VINS-Fusion’s 70.69m, even with highly dynamic and signal-blocked GPS.
• The actor-critic agent reduced error by learning optimal sensor weights in real time, with training converging at epoch 3 and achieving peak test reward.
• The average time for weight adjustment per keyframe was only 5.56ms, confirming the method’s real-time feasibility.
• The segmented global optimization effectively mitigated drift in urban scenes, even when GPS signals were corrupted by multipath and blockage.
• The framework is open-source and highly scalable, supporting easy integration of new sensor modalities through the factor graph architecture.