[Paper Review] PLVS: A SLAM System with Points, Lines, Volumetric Mapping, and 3D Incremental Segmentation

Luigi Freda|arXiv (Cornell University)|Sep 19, 2023

Advanced Vision and ImagingComputer Science3 citations

TL;DR

PLVS is a real-time, open-source SLAM system that integrates sparse keypoint and keyline tracking with volumetric mapping and 3D incremental segmentation for RGB-D and stereo cameras. It introduces a novel depth-aware reprojection error for line segments and enables geometry-based, unsupervised segmentation of volumetric maps, achieving robust, dense 3D reconstructions on CPU for small robotic platforms.

ABSTRACT

This document presents PLVS: a real-time system that leverages sparse SLAM, volumetric mapping, and 3D unsupervised incremental segmentation. PLVS stands for Points, Lines, Volumetric mapping, and Segmentation. It supports RGB-D and Stereo cameras, which may be optionally equipped with IMUs. The SLAM module is keyframe-based, and extracts and tracks sparse points and line segments as features. Volumetric mapping runs in parallel with respect to the SLAM front-end and generates a 3D reconstruction of the explored environment by fusing point clouds backprojected from keyframes. Different volumetric mapping methods are supported and integrated in PLVS. We use a novel reprojection error to bundle-adjust line segments. This error exploits available depth information to stabilize the position estimates of line segment endpoints. An incremental and geometric-based segmentation method is implemented and integrated for RGB-D cameras in the PLVS framework. We present qualitative and quantitative evaluations of the PLVS framework on some publicly available datasets. The appendix details the adopted stereo line triangulation method and provides a derivation of the Jacobians we used for line error terms. The software is available as open-source.

Motivation & Objective

• To develop a robust, real-time SLAM system that supports dense 3D reconstruction on resource-constrained robotic platforms.
• To integrate sparse SLAM with volumetric mapping and incremental 3D segmentation for improved environmental understanding.
• To enhance line segment tracking accuracy by introducing a depth-aware reprojection error that leverages depth information to stabilize endpoint estimates.
• To enable unsupervised, geometry-based 3D segmentation of volumetric maps using line segment features from RGB-D data.
• To ensure full CPU operation with optional GPU acceleration for keypoint extraction, maximizing portability on small robots.

Proposed method

• PLVS extends ORB-SLAM2 and ORB-SLAM3 with joint keypoint and keyline tracking, using a keyframe-based architecture for camera localization and map building.
• It implements a novel reprojection error for 3D line segments that incorporates depth information to stabilize endpoint position estimates, improving robustness in bundle adjustment.
• The system fuses backprojected point clouds from keyframes into a volumetric map using multiple methods: octrees, octomap, and spatially hashed voxels with Truncated Signed Distance Fields (TSDFs).
• An incremental, unsupervised segmentation method is applied to the volumetric map using geometric features from detected line segments, enabling dynamic scene understanding.
• The framework maintains consistency between sparse and volumetric maps by re-building or adjusting the volumetric map after global SLAM optimizations, such as loop closure.
• All components are modular and configurable, allowing users to trade-off map resolution and accuracy against CPU load.

Experimental results

Research questions

• RQ1 How can line segment tracking in SLAM be improved using depth information to enhance 3D reconstruction accuracy?
• RQ2 To what extent can volumetric mapping be effectively integrated with sparse SLAM to produce consistent, dense 3D maps?
• RQ3 Can an incremental, unsupervised segmentation method based on geometric features produce meaningful object-level scene understanding in real time?
• RQ4 How does the integration of points, lines, and volumetric data improve robustness and accuracy in real-time SLAM on CPU-only platforms?
• RQ5 What is the performance trade-off between map quality and computational load when enabling or disabling specific components in the system?

Key findings

• PLVS achieves real-time performance on CPU, making it suitable for deployment on small robotic platforms without dedicated GPU hardware.
• The proposed depth-aware reprojection error for line segments significantly improves endpoint stability and tracking accuracy by leveraging depth information during bundle adjustment.
• The system supports multiple volumetric mapping methods—octrees, octomap, and TSDF-based spatial hashing—offering flexibility in map resolution and memory usage.
• The incremental, geometry-based segmentation method successfully identifies and separates planar and structural elements in the 3D map using line segment features from RGB-D data.
• The framework maintains consistency between sparse and volumetric maps through synchronization with SLAM’s global optimization, ensuring accurate and coherent reconstruction.
• The open-source implementation on GitHub enables reproducibility and community extension, with support for both RGB-D and stereo cameras, including IMU-equipped systems.