[Paper Review] MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing

Jingwei Song, Jun Wang+3 more|arXiv (Cornell University)|Mar 6, 2018

Robotics and Sensor-Based Localization17 references3 citations

TL;DR

MIS-SLAM proposes a real-time, large-scale dense deformable SLAM system for minimally invasive surgery using heterogeneous CPU-GPU computing. By integrating ORB-SLAM on CPU for robust global pose estimation with GPU-accelerated dense reconstruction and fusion, it achieves real-time, accurate 3D mapping of deformable tissues despite fast camera motion and image blur, demonstrating feasibility on in-vivo datasets with sub-millimeter accuracy.

ABSTRACT

Real-time simultaneously localization and dense mapping is very helpful for providing Virtual Reality and Augmented Reality for surgeons or even surgical robots. In this paper, we propose MIS-SLAM: a complete real-time large scale dense deformable SLAM system with stereoscope in Minimal Invasive Surgery based on heterogeneous computing by making full use of CPU and GPU. Idled CPU is used to perform ORB- SLAM for providing robust global pose. Strategies are taken to integrate modules from CPU and GPU. We solved the key problem raised in previous work, that is, fast movement of scope and blurry images make the scope tracking fail. Benefiting from improved localization, MIS-SLAM can achieve large scale scope localizing and dense mapping in real-time. It transforms and deforms current model and incrementally fuses new observation while keeping vivid texture. In-vivo experiments conducted on publicly available datasets presented in the form of videos demonstrate the feasibility and practicality of MIS-SLAM for potential clinical purpose.

Motivation & Objective

• To address the instability in scope tracking caused by fast movement and image blur during minimally invasive surgery.
• To develop a real-time, large-scale dense deformable SLAM system that enables accurate 3D reconstruction of soft tissues.
• To integrate CPU-based ORB-SLAM for global pose estimation with GPU-accelerated dense mapping for improved robustness and performance.
• To enable practical clinical applications of AR/VR in surgery through real-time, high-fidelity tissue modeling.

Proposed method

• Utilizes a heterogeneous computing architecture with CPU running an improved ORB-SLAM for robust global pose estimation and feature tracking.
• Transfers ORB features and global pose from CPU to GPU global memory for fusion with dense GPU-based reconstruction.
• Employs GPU-accelerated depth estimation, non-rigid registration, and incremental fusion using a novel point cloud management strategy to reduce memory usage and improve performance.
• Applies a warping field based on embedded deformation nodes to model tissue deformation from stereo image pairs.
• Uses a first-in-last-out stack data structure to prioritize the latest data on GPU, ensuring real-time processing despite GPU latency.
• Visualizes the reconstructed 3D model directly on GPU using CUDA, minimizing data transfer overhead.

Experimental results

Research questions

• RQ1 Can a heterogeneous CPU-GPU architecture enable real-time dense deformable SLAM in minimally invasive surgery despite fast camera motion and image blur?
• RQ2 How can ORB-SLAM's robust global pose estimation be effectively integrated with GPU-accelerated dense reconstruction to improve system stability?
• RQ3 What fusion strategy can maintain high accuracy and low memory usage during large-scale, long-duration soft-tissue reconstruction?
• RQ4 How does global pose initialization impact back-projection accuracy in dynamic, non-rigid surgical environments?
• RQ5 Can the system achieve real-time visualization and reconstruction of deformable tissues with clinically acceptable accuracy?

Key findings

• MIS-SLAM achieves real-time processing at approximately 0.07 seconds per frame on a single desktop using an NVIDIA GeForce GTX TITAN X.
• The system demonstrates sub-millimeter accuracy, with average back-projection errors of 0.41 mm (heart), 0.66 mm (liver), and 0.62 mm (right kidney) when pose initialization is used.
• Without pose initialization, average errors increase to 0.46 mm, 0.68 mm, and 0.82 mm respectively, confirming the importance of initialization for accuracy.
• Ex-vivo experiments on Hamlyn phantom datasets show average accuracies of 0.28 mm and 0.35 mm, validating the system's precision in controlled settings.
• The system successfully handles large-scale reconstruction and maintains visual quality despite challenges like illumination differences, texture blending issues, and motion blur.
• The use of a point cloud management strategy significantly reduces memory usage compared to traditional TSDF-based fusion, enhancing scalability.