[Paper Review] SemGauss-SLAM: Dense Semantic Gaussian Splatting SLAM

Siting Zhu, Renjie Qin+3 more|arXiv (Cornell University)|Mar 12, 2024

Robotics and Sensor-Based Localization4 citations

TL;DR

SemGauss-SLAM is the first 3D Gaussian-based semantic SLAM system that integrates semantic feature embedding into 3D Gaussians for real-time dense semantic mapping, robust camera tracking, and high-fidelity rendering. It introduces feature-level loss for semantic optimization and semantic-informed bundle adjustment to reduce drift, achieving state-of-the-art performance on Replica and ScanNet in tracking, reconstruction, and semantic segmentation accuracy.

ABSTRACT

We propose SemGauss-SLAM, a dense semantic SLAM system utilizing 3D Gaussian representation, that enables accurate 3D semantic mapping, robust camera tracking, and high-quality rendering simultaneously. In this system, we incorporate semantic feature embedding into 3D Gaussian representation, which effectively encodes semantic information within the spatial layout of the environment for precise semantic scene representation. Furthermore, we propose feature-level loss for updating 3D Gaussian representation, enabling higher-level guidance for 3D Gaussian optimization. In addition, to reduce cumulative drift in tracking and improve semantic reconstruction accuracy, we introduce semantic-informed bundle adjustment. By leveraging multi-frame semantic associations, this strategy enables joint optimization of 3D Gaussian representation and camera poses, resulting in low-drift tracking and accurate semantic mapping. Our SemGauss-SLAM demonstrates superior performance over existing radiance field-based SLAM methods in terms of mapping and tracking accuracy on Replica and ScanNet datasets, while also showing excellent capabilities in high-precision semantic segmentation and dense semantic mapping.

Motivation & Objective

• To develop a real-time dense semantic SLAM system that overcomes limitations of existing methods in semantic accuracy and drift accumulation.
• To enable unbounded, efficient 3D semantic mapping from 2D semantic embeddings without predefined scene bounds.
• To improve tracking robustness and mapping consistency by leveraging multi-view semantic consistency in optimization.
• To achieve high-quality semantic novel-view synthesis and photo-realistic reconstruction using 3D Gaussian representation.
• To reduce cumulative drift in SLAM through semantic-informed bundle adjustment that jointly optimizes poses and scene representation.

Proposed method

• Incorporates semantic feature embeddings directly into 3D Gaussian representations to encode semantic information within the spatial layout of the scene.
• Introduces a feature-level loss function that guides 3D Gaussian optimization using high-level semantic features, accelerating convergence and improving detail fidelity.
• Proposes semantic-informed bundle adjustment that jointly optimizes camera poses and 3D Gaussian representations using multi-view semantic consistency as constraints.
• Leverages co-visible frames to establish semantic associations, enabling robust optimization even in sparsely observed regions.
• Uses 3D Gaussian splatting for efficient, high-quality rendering and integrates semantic supervision at the feature level to enhance scene representation.
• Employs a two-stage optimization pipeline: initial 3D transfer of 2D semantic features via direct propagation, followed by joint refinement using geometric, photometric, and semantic constraints.

Experimental results

Research questions

• RQ1 Can 3D Gaussian representation be effectively extended to support dense, real-time semantic SLAM with accurate 3D semantic mapping?
• RQ2 How can semantic feature embedding be efficiently propagated from 2D to 3D space to enable unbounded and fast semantic optimization?
• RQ3 Can feature-level loss improve semantic segmentation accuracy and convergence speed in 3D Gaussian-based scene representation?
• RQ4 To what extent can semantic-informed bundle adjustment reduce cumulative tracking drift compared to traditional geometric-only BA?
• RQ5 How does the integration of semantic constraints enhance novel-view semantic synthesis and reconstruction quality?

Key findings

• SemGauss-SLAM achieves state-of-the-art mIoU of 92.81 on the Replica dataset, outperforming SNI-SLAM by 9.21 percentage points.
• The method reduces RMSE to 0.26 and Depth L1 to 0.54 on the room0 scene, demonstrating superior geometric reconstruction accuracy.
• Feature-level loss improves mIoU by 2.71 points on average across three Replica scenes, with notable gains in boundary and small-object segmentation.
• Semantic-informed bundle adjustment reduces RMSE by 0.09 and Depth L1 by 0.14 compared to baseline without semantic constraints, highlighting its impact on tracking and geometry.
• The ablation study confirms that semantic constraints are more effective than RGB or depth constraints alone in reducing drift and improving semantic precision.
• Qualitative results show significant improvement in semantic novel-view synthesis, especially for poorly observed regions like ceilings, which are often misclassified in baseline methods.