[Paper Review] Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting

Yiming Huang, Beilei Cui+5 more|arXiv (Cornell University)|Jan 29, 2024

Ophthalmology and Visual Impairment StudiesMedicine3 citations

TL;DR

Endo-4DGS proposes a real-time, monocular endoscopic scene reconstruction method using 4D Gaussian Splatting and Depth-Anything for pseudo-depth estimation, enabling high-fidelity dynamic tissue reconstruction without ground truth depth. It achieves 100 FPS inference, 4-minute training, and 4GB GPU usage—surpassing state-of-the-art methods in accuracy and efficiency on two surgical datasets.

ABSTRACT

In the realm of robot-assisted minimally invasive surgery, dynamic scene reconstruction can significantly enhance downstream tasks and improve surgical outcomes. Neural Radiance Fields (NeRF)-based methods have recently risen to prominence for their exceptional ability to reconstruct scenes but are hampered by slow inference speed, prolonged training, and inconsistent depth estimation. Some previous work utilizes ground truth depth for optimization but is hard to acquire in the surgical domain. To overcome these obstacles, we present Endo-4DGS, a real-time endoscopic dynamic reconstruction approach that utilizes 3D Gaussian Splatting (GS) for 3D representation. Specifically, we propose lightweight MLPs to capture temporal dynamics with Gaussian deformation fields. To obtain a satisfactory Gaussian Initialization, we exploit a powerful depth estimation foundation model, Depth-Anything, to generate pseudo-depth maps as a geometry prior. We additionally propose confidence-guided learning to tackle the ill-pose problems in monocular depth estimation and enhance the depth-guided reconstruction with surface normal constraints and depth regularization. Our approach has been validated on two surgical datasets, where it can effectively render in real-time, compute efficiently, and reconstruct with remarkable accuracy.

Motivation & Objective

• Address the limitations of NeRF-based methods in endoscopic reconstruction, including slow inference, high computational cost, and reliance on stereo depth.
• Overcome the infeasibility of stereo cameras in minimally invasive surgery due to size, cost, and logistical constraints.
• Enable accurate, real-time reconstruction of deformable endoscopic scenes using only monocular video input.
• Reduce training time and GPU memory consumption while maintaining high reconstruction fidelity for clinical deployment.
• Integrate a lightweight MLP to model temporal dynamics of tissue deformation, enhancing reconstruction of dynamic surgical environments.

Proposed method

• Extend 3D Gaussian Splatting to 4D by incorporating time as a fourth dimension, enabling spatiotemporal modeling of dynamic endoscopic scenes.
• Use Depth-Anything, a vision foundation model, to generate pseudo-depth maps from monocular endoscopic images, providing depth supervision without ground truth data.
• Introduce a lightweight MLP to predict temporal deformations of 4D Gaussians, modeling tissue motion and shape changes over time.
• Optimize 4D Gaussians using a differentiable rasterizer with spatial, angular, and alpha blending properties for high-fidelity rendering.
• Leverage a depth ranking loss with hyperparameters α=1000 and λ=0.01 to improve pseudo-depth quality and reconstruction consistency.
• Train the model end-to-end using a voxelized 4D grid of size [64,64,64,75] with Adam optimizer and a learning rate of 1.6×10⁻³.

Experimental results

Research questions

• RQ1 Can 4D Gaussian Splatting be effectively adapted to monocular endoscopic scene reconstruction without ground truth depth?
• RQ2 How well can a vision foundation model like Depth-Anything generalize to endoscopic images for pseudo-depth estimation?
• RQ3 Can a lightweight MLP effectively model tissue deformation dynamics in real-time surgical scenes?
• RQ4 Does the proposed method achieve real-time inference while maintaining high reconstruction accuracy compared to NeRF-based baselines?
• RQ5 To what extent does the method reduce training time and GPU memory usage compared to existing state-of-the-art approaches?

Key findings

• Endo-4DGS achieves a PSNR of 36.84 and SSIM of 0.954 on the StereoMIS dataset, outperforming EndoNeRF (21.49 PSNR) and EndoSurf (34.89 PSNR).
• On the EndoNeRF dataset, Endo-4DGS achieves 37.08 PSNR and 0.955 SSIM, significantly exceeding EndoNeRF (21.49 PSNR) and EndoSurf (34.91 PSNR).
• The method attains 100 FPS inference speed, a substantial improvement over EndoNeRF (0.2 FPS) and EndoSurf (0.04 FPS), enabling real-time application.
• Training time is reduced to just 4 minutes, compared to 7 hours for EndoSurf and 5 hours for EndoNeRF, with only 4GB GPU memory usage.
• The method demonstrates superior generalization on challenging scenes with tool occlusion and non-rigid tissue deformation, as shown in qualitative comparisons.
• The integration of Depth-Anything enables high-quality pseudo-depth estimation, supporting accurate depth-guided reconstruction without ground truth depth.