[Paper Review] Augmented Reality for Depth Cues in Monocular Minimally Invasive Surgery

Long Chen, Wen Tang+3 more|arXiv (Cornell University)|Mar 1, 2017

Augmented Reality Applications5 references8 citations

TL;DR

This paper proposes a real-time 3D surface reconstruction and augmented reality framework for monocular laparoscopy using ORB-SLAM for camera tracking and a robust pipeline combining outlier filtering, Moving Least Squares, and Poisson surface reconstruction. It achieves 1.24 mm RMSE in camera trajectory estimation and 4.32 mm RMSD in surface reconstruction, enabling depth-correct AR visualization for improved depth perception in minimally invasive surgery.

ABSTRACT

One of the major challenges in Minimally Invasive Surgery (MIS) such as laparoscopy is the lack of depth perception. In recent years, laparoscopic scene tracking and surface reconstruction has been a focus of investigation to provide rich additional information to aid the surgical process and compensate for the depth perception issue. However, robust 3D surface reconstruction and augmented reality with depth perception on the reconstructed scene are yet to be reported. This paper presents our work in this area. First, we adopt a state-of-the-art visual simultaneous localization and mapping (SLAM) framework - ORB-SLAM - and extend the algorithm for use in MIS scenes for reliable endoscopic camera tracking and salient point mapping. We then develop a robust global 3D surface reconstruction frame- work based on the sparse point clouds extracted from the SLAM framework. Our approach is to combine an outlier removal filter within a Moving Least Squares smoothing algorithm and then employ Poisson surface reconstruction to obtain smooth surfaces from the unstructured sparse point cloud. Our proposed method has been quantitatively evaluated compared with ground-truth camera trajectories and the organ model surface we used to render the synthetic simulation videos. In vivo laparoscopic videos used in the tests have demonstrated the robustness and accuracy of our proposed framework on both camera tracking and surface reconstruction, illustrating the potential of our algorithm for depth augmentation and depth-corrected augmented reality in MIS with monocular endoscopes.

Motivation & Objective

• To address the critical challenge of depth perception loss in monocular minimally invasive surgery (MIS) due to 2D endoscopic views.
• To enable real-time, accurate 3D reconstruction of laparoscopic scenes from monocular endoscope video for depth augmentation.
• To develop a depth-correct augmented reality system that superimposes 3D anatomical models at correct spatial positions relative to tissue surfaces.
• To overcome limitations of stereo endoscopy, such as cost and limited field of view, by using visual SLAM for global scene understanding.
• To validate the framework on both synthetic simulation videos and real in vivo laparoscopic data for clinical relevance.

Proposed method

• Adapted ORB-SLAM for monocular MIS by extending it to handle rapid motion, luminance changes, and soft tissue deformation in endoscopic scenes.
• Extracted sparse 3D point clouds from ORB-SLAM's feature tracking and map point estimation for scene reconstruction.
• Applied an outlier removal filter followed by Moving Least Squares (MLS) smoothing to denoise and regularize the sparse point cloud.
• Employed Poisson surface reconstruction to generate smooth, watertight 3D surfaces from the preprocessed point cloud.
• Used a simple Iterative Closest Point (ICP) algorithm to align the reconstructed surface with the ground truth 3D model for quantitative evaluation.
• Integrated camera pose and 3D surface data into a real-time augmented reality system to superimpose depth-correct 3D annotations and models onto the endoscopic video stream.

Experimental results

Research questions

• RQ1 Can ORB-SLAM be effectively adapted for reliable real-time camera tracking in challenging monocular laparoscopic environments?
• RQ2 Can a robust pipeline combining outlier filtering, MLS smoothing, and Poisson reconstruction produce accurate 3D surfaces from sparse, noisy point clouds in MIS?
• RQ3 To what extent does the reconstructed 3D surface match the ground truth in terms of geometric accuracy?
• RQ4 Can depth-correct augmented reality be achieved using the reconstructed 3D surface and camera pose for improved depth perception in monocular MIS?
• RQ5 How does the framework perform on real in vivo laparoscopic video data compared to synthetic simulations?

Key findings

• The ORB-SLAM-based camera tracking achieved a root mean square error (RMSE) of 1.24 mm when compared to ground truth camera trajectories in synthetic simulations.
• The 3D surface reconstruction achieved an RMSD of 4.32 mm compared to the ground truth 3D model, indicating strong geometric fidelity.
• The reconstructed 3D surface closely aligned with the ground truth model, as shown in visual comparisons and distance maps, with most surface distances falling between -1.0 mm and 4.0 mm.
• The framework successfully enabled real-time depth-correct augmented reality, with virtual 3D annotations correctly positioned in depth relative to the tissue surface.
• The system demonstrated robustness and accuracy on real in vivo laparoscopic video sequences, validating its potential for intra-operative use.
• The entire surface reconstruction pipeline completed in 600 ms, enabling real-time performance suitable for clinical deployment.