[Paper Review] SD-DefSLAM: Semi-Direct Monocular SLAM for Deformable and Intracorporeal Scenes

Juan J. Gómez Rodríguez, José Lamarca+3 more|arXiv (Cornell University)|Oct 19, 2020

Robotics and Sensor-Based Localization30 references4 citations

TL;DR

SD-DefSLAM is a semi-direct monocular SLAM system designed for highly deformable and intracorporeal environments, combining illumination-invariant photometric tracking with CNN-based surgical tool segmentation and bag-of-words relocalization. It significantly reduces scale drift and improves reconstruction accuracy over DefSLAM, becoming the first system to robustly perform SLAM inside the human body under extreme deformations and poor texture conditions.

ABSTRACT

Conventional SLAM techniques strongly rely on scene rigidity to solve data association, ignoring dynamic parts of the scene. In this work we present Semi-Direct DefSLAM (SD-DefSLAM), a novel monocular deformable SLAM method able to map highly deforming environments, built on top of DefSLAM. To robustly solve data association in challenging deforming scenes, SD-DefSLAM combines direct and indirect methods: an enhanced illumination-invariant Lucas-Kanade tracker for data association, geometric Bundle Adjustment for pose and deformable map estimation, and bag-of-words based on feature descriptors for camera relocation. Dynamic objects are detected and segmented-out using a CNN trained for the specific application domain. We thoroughly evaluate our system in two public datasets. The mandala dataset is a SLAM benchmark with increasingly aggressive deformations. The Hamlyn dataset contains intracorporeal sequences that pose serious real-life challenges beyond deformation like weak texture, specular reflections, surgical tools and occlusions. Our results show that SD-DefSLAM outperforms DefSLAM in point tracking, reconstruction accuracy and scale drift thanks to the improvement in all the data association steps, being the first system able to robustly perform SLAM inside the human body.

Motivation & Objective

• Address the limitations of conventional SLAM in highly deforming, intracorporeal environments such as those in minimally invasive surgery.
• Overcome the rigidity assumption in SLAM by enabling accurate data association and 3D map estimation under isometric deformations.
• Improve robustness to weak texture, specular reflections, occlusions, and dynamic surgical tools common in endoscopic sequences.
• Integrate relocalization and dynamic object segmentation to handle tracking failures and maintain long-term consistency.
• Develop a semi-direct approach that combines direct photometric optimization with indirect feature-based tracking for enhanced data association.

Proposed method

• Uses per-feature, illumination-invariant Lucas-Kanade optical flow for subpixel-accurate photometric data association, replacing global image alignment.
• Applies geometric Bundle Adjustment to jointly optimize camera pose and deformable 3D map, minimizing reprojection and photometric errors.
• Employs ORB features for long-term data association and bag-of-words relocalization to recover from complete camera occlusions.
• Integrates a CNN trained on surgical tool segmentation to mask out dynamic objects and prevent them from corrupting the map.
• Implements active tracking of feature patches across frames, eliminating reliance on feature extraction in the matching stage.
• Assumes isometric deformations in the 3D map, modeling time-varying landmark positions as smooth, non-rigid transformations.

Experimental results

Research questions

• RQ1 Can a semi-direct SLAM system achieve robust data association in highly deforming, low-texture intracorporeal scenes where rigid SLAM fails?
• RQ2 How does per-feature illumination-invariant photometric tracking compare to conventional feature matching in terms of tracking accuracy and robustness under deformation?
• RQ3 To what extent does CNN-based surgical tool segmentation improve SLAM performance in endoscopic sequences with dynamic objects?
• RQ4 Can relocalization with bag-of-words and ORB descriptors restore tracking after complete occlusion in real-time endoscopic sequences?
• RQ5 Does combining direct photometric optimization with indirect feature tracking reduce scale drift and improve reconstruction accuracy in non-rigid environments?

Key findings

• SD-DefSLAM achieves significantly lower scale drift and better RMSE than DefSLAM across all sequences in the Mandala and Hamlyn datasets.
• In the Mandala3 sequence, SD-DefSLAM doubles the percentage of correctly matched map points compared to DefSLAM, improving data association robustness.
• On Hamlyn Dataset20, which features low texture and blurring, SD-DefSLAM maintains superior performance due to subpixel-accurate photometric tracking.
• The integration of CNN-based tool segmentation in Sequence_heart and Sequence_organs enables robust processing of sequences with high dynamic object interference.
• The system successfully performs relocalization after complete occlusion or endoscope removal, as demonstrated in Dataset19 and Dataset20.
• 3D reconstructions show accurate modeling of deformations induced by surgical tools, as visualized in Sequence_organs with frames #315, #1010, #1030, and #1055.