[Paper Review] Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splatting

Vladimir Yugay, Yue Li+2 more|arXiv (Cornell University)|Dec 6, 2023

Robotics and Sensor-Based Localization18 citations

TL;DR

Gaussian-SLAM introduces a dense RGBD SLAM system that uses 3D Gaussian splats as a scene representation to achieve photo-realistic rendering at interactive speeds, with online sub-map management and geometry encoding. It delivers state-of-the-art rendering quality and competitive reconstruction on real-world data.

ABSTRACT

We present a dense simultaneous localization and mapping (SLAM) method that uses 3D Gaussians as a scene representation. Our approach enables interactive-time reconstruction and photo-realistic rendering from real-world single-camera RGBD videos. To this end, we propose a novel effective strategy for seeding new Gaussians for newly explored areas and their effective online optimization that is independent of the scene size and thus scalable to larger scenes. This is achieved by organizing the scene into sub-maps which are independently optimized and do not need to be kept in memory. We further accomplish frame-to-model camera tracking by minimizing photometric and geometric losses between the input and rendered frames. The Gaussian representation allows for high-quality photo-realistic real-time rendering of real-world scenes. Evaluation on synthetic and real-world datasets demonstrates competitive or superior performance in mapping, tracking, and rendering compared to existing neural dense SLAM methods.

Motivation & Objective

• Motivate dense SLAM with a Gaussian splatting scene representation for high-fidelity rendering.
• Extend Gaussian splatting from offline multi-view to online monocular RGBD SLAM.
• Encode geometry within Gaussian splats to improve 3D reconstruction in monocular setups.
• Develop online sub-map seeding and optimization strategies to maintain interactive performance.
• Investigate frame-to-model tracking using the Gaussian scene representation and compare to frame-to-frame tracking.

Proposed method

• Represent the scene with a set of 3D Gaussians with parameters (mean, scale, rotation, opacity, spherical harmonics).
• Split the input sequence into sub-maps to enable online learning and prevent catastrophic forgetting; optimize the active sub-map with depth and color losses.
• Seed new Gaussians from dense point clouds in keyframes and anchor them behind surfaces in viewing direction to initialize geometry.
• Render using a differentiable rasterizer with color and depth losses combining L1 and SSIM for color and an L1 depth loss, plus a regularization term to prevent scale explosion.
• Track by initializing pose with RGBD odometry and refining frame-to-model re-rendering loss, while noting extrapolation limitations of Gaussian splats.

Experimental results

Research questions

• RQ1 Can Gaussian splats be effectively extended to encode geometry for online monocular SLAM with RGBD input?
• RQ2 How can online sub-map seeding and optimization be designed to maintain interactive performance without catastrophic forgetting?
• RQ3 What is the impact of frame-to-model tracking using Gaussian splats compared to frame-to-frame tracking in dense SLAM?
• RQ4 What limits do Gaussian splats impose on geometry accuracy and extrapolation, and can these be mitigated in SLAM?

Key findings

• Gaussian-SLAM achieves state-of-the-art rendering quality on ScanNet and comparable reconstruction performance to dense neural SLAM methods.
• On ScanNet, Gaussian-SLAM attains PSNR 37.45, SSIM 0.984, LPIPS 0.068, outperforming NICE-SLAM, Vox-Fusion, ESLAM, and Point-SLAM in rendering metrics.
• On TUM-RGBD, Gaussian-SLAM again shows strong rendering metrics with substantial improvements over competing methods (Table 2 results).
• Sub-map based online seeding and optimization enable interactive-time reconstruction while preserving geometry obtained from depth sensors.
• Tracking with Gaussian splats shows limitations in frame-to-model tracking due to extrapolation; oracle experiments indicate potential improvements with better depth rendering.