[Paper Review] SLAM Endoscopy enhanced by adversarial depth prediction

Richard J. Chen, Taylor L. Bobrow+3 more|arXiv (Cornell University)|Jun 29, 2019

Colorectal Cancer Screening and DetectionMedicine13 references21 citations

TL;DR

This paper proposes a monocular SLAM system for endoscopy enhanced by adversarial depth prediction using a conditional GAN to estimate depth from RGB endoscopic images. Trained on synthetic and domain-randomized photorealistic CT-rendered colon images, the method enables dense 3D reconstruction of phantom and ex-vivo porcine colon models, achieving robust tracking despite specular reflections and feature sparsity.

ABSTRACT

Medical endoscopy remains a challenging application for simultaneous localization and mapping (SLAM) due to the sparsity of image features and size constraints that prevent direct depth-sensing. We present a SLAM approach that incorporates depth predictions made by an adversarially-trained convolutional neural network (CNN) applied to monocular endoscopy images. The depth network is trained with synthetic images of a simple colon model, and then fine-tuned with domain-randomized, photorealistic images rendered from computed tomography measurements of human colons. Each image is paired with an error-free depth map for supervised adversarial learning. Monocular RGB images are then fused with corresponding depth predictions, enabling dense reconstruction and mosaicing as an endoscope is advanced through the gastrointestinal tract. Our preliminary results demonstrate that incorporating monocular depth estimation into a SLAM architecture can enable dense reconstruction of endoscopic scenes.

Motivation & Objective

• Address the challenge of sparse features and lack of depth sensing in monocular endoscopic SLAM for accurate 3D reconstruction.
• Overcome limitations of traditional feature-based SLAM in endoscopy due to tissue homogeneity, deformation, and specular reflections.
• Develop a depth estimation method that generalizes across patient-specific textures and colors using adversarial training.
• Enable dense surfel-based reconstruction of the gastrointestinal tract using only monocular RGB video and predicted depth.
• Provide a framework for mosaicing endoscopic scenes to improve procedural quality metrics and lesion localization.

Proposed method

• Train a conditional Generative Adversarial Network (cGAN) to predict depth from monocular RGB endoscopy images using a min-max loss objective.
• Use synthetic, cinematic renderings of a simple colon model as initial training data with ground-truth depth maps.
• Fine-tune the depth network on domain-randomized, photorealistic images generated from CT scans of human colons.
• Apply adversarial loss (L_GAN) combined with L1 pixel-wise loss to improve perceptual realism and accuracy of depth predictions.
• Fuse predicted depth maps with RGB images into the ElasticFusion SLAM framework for direct, dense surfel-based reconstruction.
• Utilize a monocular camera setup without hardware modifications, enabling real-time, robust tracking in challenging endoscopic conditions.

Experimental results

Research questions

• RQ1 Can adversarial depth prediction from monocular endoscopy images improve 3D reconstruction accuracy in feature-scarce, specular, and deformable environments?
• RQ2 How effective is domain randomization in enhancing generalization of depth estimation networks to real endoscopic tissue?
• RQ3 To what extent can a GAN-based depth network handle variations in tissue color, texture, and specular reflections in the GI tract?
• RQ4 Can dense surfel-based reconstruction be achieved using only monocular RGB and predicted depth, without stereo or depth sensors?
• RQ5 How does the proposed method compare in tracking stability and reconstruction quality to conventional feature-based SLAM in endoscopy?

Key findings

• The proposed method achieved dense 3D reconstruction of a phantom colon model, accurately capturing haustral folds and fiducial markers.
• In ex-vivo porcine colon models, the system successfully tracked and recolored surfels despite dynamic changes in illumination and specular reflections.
• The adversarial depth network demonstrated robustness to texture and color variations through domain randomization during fine-tuning.
• The framework enabled mosaicing of endoscopic video sequences into a coherent 3D surface model without hardware modifications.
• The reconstruction quality was qualitatively validated on both phantom and real tissue models, with quantitative validation pending.
• The approach supports potential clinical applications such as improved lesion localization and assessment of procedural completeness (e.g., surface area coverage).