[Paper Review] Structure PLP-SLAM: Efficient Sparse Mapping and Localization using Point, Line and Plane for Monocular, RGB-D and Stereo Cameras

Fangwen Shu, Jiaxuan Wang+2 more|arXiv (Cornell University)|Jul 13, 2022

Robotics and Sensor-Based Localization4 citations

TL;DR

This paper proposes Structure PLP-SLAM, a monocular, RGB-D, and stereo visual SLAM system that integrates points, lines, and planes for robust camera localization and real-time structural mapping. By using Plücker coordinates for 3D lines and adaptive plane fitting, it achieves state-of-the-art trajectory accuracy on TUM RGB-D, ICL-NUIM, and EuRoC datasets, outperforming existing methods in both precision and semantic map quality.

ABSTRACT

This paper presents a visual SLAM system that uses both points and lines for robust camera localization, and simultaneously performs a piece-wise planar reconstruction (PPR) of the environment to provide a structural map in real-time. One of the biggest challenges in parallel tracking and mapping with a monocular camera is to keep the scale consistent when reconstructing the geometric primitives. This further introduces difficulties in graph optimization of the bundle adjustment (BA) step. We solve these problems by proposing several run-time optimizations on the reconstructed lines and planes. Our system is able to run with depth and stereo sensors in addition to the monocular setting. Our proposed SLAM tightly incorporates the semantic and geometric features to boost both frontend pose tracking and backend map optimization. We evaluate our system exhaustively on various datasets, and show that we outperform state-of-the-art methods in terms of trajectory precision. The code of PLP-SLAM has been made available in open-source for the research community (https://github.com/PeterFWS/Structure-PLP-SLAM).

Motivation & Objective

• To address scale ambiguity and inconsistent geometric optimization in monocular SLAM by integrating higher-level geometric features.
• To improve camera localization robustness in low-texture and variable lighting conditions using lines and planes alongside points.
• To enable real-time, piecewise planar reconstruction without strong assumptions like Manhattan World.
• To develop a sensor-agnostic SLAM framework that supports monocular, RGB-D, and stereo inputs.
• To provide a re-localization module based on a pre-built point-line map for robust recovery after tracking failure.

Proposed method

• Represents 3D lines using Plücker coordinates for minimal, unambiguous parameterization, while maintaining endpoint representations for visualization and correspondence search.
• Uses a CNN-based instance planar segmentation network to generate initial plane priors, followed by RANSAC-based graph optimization for robust 3D plane fitting.
• Implements a modular, multi-feature SLAM pipeline based on OpenVSLAM, extending ORB-SLAM2 with joint optimization of points, lines, and planes in bundle adjustment.
• Applies adaptive thresholding in RANSAC to reduce sensitivity to noise and improve plane fitting reliability without strong assumptions.
• Integrates loop closure and re-localization using a pre-built point-line map, with pose optimization via motion-only BA using 3D-2D point and line correspondences.
• Employs a map refiner to retain only high-quality planes, improving map consistency and reducing noise propagation.

Figure 1: The proposed SLAM system utilizes point and line cloud for robust camera localization, with piece-wise planar reconstruction for semantic mapping. Here, the reconstructed maps are presented selectively for better visualization.

Experimental results

Research questions

• RQ1 Can the integration of lines and planes with points significantly improve camera localization robustness in challenging visual conditions?
• RQ2 How can 3D lines be represented and optimized efficiently in a graph-based SLAM framework without over-parameterization?
• RQ3 Can real-time, piecewise planar reconstruction be achieved robustly in monocular SLAM without relying on strong geometric assumptions like Manhattan World?
• RQ4 Does the use of a pre-built point-line map enhance re-localization performance compared to point-only methods?
• RQ5 How does the proposed system compare quantitatively to state-of-the-art SLAM systems across diverse sensor modalities and datasets?

Key findings

• On the TUM RGB-D dataset, Structure PLP-SLAM achieved an average ATE RMSE of 1.87 cm with full point-line-plane integration, outperforming SP-SLAM (1.51 cm) and Manhattan SLAM (2.81 cm).
• On the ICL-NUIM dataset, the system achieved an average ATE RMSE of 1.34 cm with point-line-plane fusion, surpassing the next best method (1.77 cm) in trajectory accuracy.
• The re-localization module using both points and lines reduced absolute pose errors (APE) by up to 30% compared to point-only optimization, as shown in the single-image localization evaluation.
• The system demonstrated robust performance across diverse scenes, including low-texture environments, due to the complementary use of lines and planes in localization and mapping.
• The map refiner successfully filtered out low-quality and small planes, improving map consistency and reducing noise, though it limited plane reconstruction in very low-texture scenes.
• The open-source release of the code on GitHub enables reproducibility and further research in semantic SLAM with multi-feature integration.

(a) Endpoints representation.