[Paper Review] LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping

Tixiao Shan, Brendan Englot+2 more|ArXiv.org|Apr 22, 2021

Robotics and Sensor-Based LocalizationEngineering25 references29 citations

TL;DR

LVI-SAM presents a tightly-coupled LVIO framework built on a factor graph with two interacting subsystems (VIS and LIS) that share information, enable robust initialization, enable loop closures, and remain functional if one subsystem fails, for real-time state estimation and mapping.

ABSTRACT

We propose a framework for tightly-coupled lidar-visual-inertial odometry via smoothing and mapping, LVI-SAM, that achieves real-time state estimation and map-building with high accuracy and robustness. LVI-SAM is built atop a factor graph and is composed of two sub-systems: a visual-inertial system (VIS) and a lidar-inertial system (LIS). The two sub-systems are designed in a tightly-coupled manner, in which the VIS leverages LIS estimation to facilitate initialization. The accuracy of the VIS is improved by extracting depth information for visual features using lidar measurements. In turn, the LIS utilizes VIS estimation for initial guesses to support scan-matching. Loop closures are first identified by the VIS and further refined by the LIS. LVI-SAM can also function when one of the two sub-systems fails, which increases its robustness in both texture-less and feature-less environments. LVI-SAM is extensively evaluated on datasets gathered from several platforms over a variety of scales and environments. Our implementation is available at https://git.io/lvi-sam

Motivation & Objective

• Improve robustness and accuracy of lidar-visual-inertial odometry by tightly coupling visual-inertial and lidar-inertial systems in a factor-graph framework.
• Enable reliable initialization, feature-depth estimation from lidar, and loop closures to reduce drift.
• Provide robustness by allowing either VIS or LIS to operate independently if the other fails.
• Validate performance across diverse datasets and platforms to establish a strong baseline for LVIO.

Proposed method

• Two sub-systems operating in a tightly-coupled fashion: a visual-inertial system (VIS) and a lidar-inertial system (LIS).
• VIS minimizes visual reprojection errors and IMU residuals, and can obtain feature depth from lidar to improve initialization and depth estimation.
• LIS performs lidar feature extraction and scan-matching against a sliding-window feature map, using IMU preintegration and an initial guess from IMU and VIS.
• A factor graph aggregates IMU preintegration, visual odometry, lidar odometry, and loop-closure constraints, solved with iSAM2 for MAP estimation.
• Loop closures are first identified by the VIS and refined by the LIS, enabling global optimization and drift reduction.
• System robustness is enhanced by failure detection that allows either VIS or LIS to re-initialize if the other subsystem fails.

Figure 1: The system structure of LVI-SAM. The system, which receives input from a 3D lidar, a camera and an IMU, can be divided into two sub-systems: a visual-inertial system (VIS) and a lidar-inertial system (LIS). The VIS and LIS can function independently while using information from each other

Experimental results

Research questions

• RQ1 Can a tightly-coupled LVIO framework leveraging both VIS and LIS improve robustness and accuracy over single-sensor or loosely-coupled approaches?
• RQ2 How can initialization, depth estimation, and loop closure be designed to operate robustly in texture-less or degraded environments?
• RQ3 What is the role of a factor-graph-based optimization in fusing VIS, LIS, IMU, and loop-closure constraints for real-time mapping?
• RQ4 Can the system sustain real-time performance while maintaining strong accuracy across varied platforms and scales?

Key findings

• The proposed framework achieves real-time state estimation and mapping with robustness in degraded environments.
• Depth-augmented visual features from lidar improve VIS initialization and overall accuracy.
• Loop-closure via the VIS is refined by the LIS, enabling global optimization and drift reduction.
• The VIS and LIS can operate independently if one subsystem fails, increasing robustness in texture-less settings.
• Extensive experiments on multiple datasets and platforms show competitive or superior accuracy compared with existing LVIO/LIO baselines.

Figure 2: The framework of our visual-inertial system. The system optimizes residuals from IMU preintegration, visual measurements without depth, and visual measurements with depth.