[Paper Review] Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration

Taylor L. Bobrow, Mayank Golhar+4 more|arXiv (Cornell University)|Jun 17, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper introduces the Colonoscopy 3D Video Dataset (C3VD), a high-fidelity benchmark for 3D computer vision in colonoscopy, created by registering real HD colonoscopy video with synthetic 3D models using a novel 2D-3D registration pipeline. The method leverages a GAN to predict depth from 2D frames and an evolutionary optimizer to align edge features, achieving 0.321 mm translation and 0.159° rotation error, enabling accurate pixel-level ground truth for depth, normals, flow, and pose across 10,015 frames.

ABSTRACT

Screening colonoscopy is an important clinical application for several 3D computer vision techniques, including depth estimation, surface reconstruction, and missing region detection. However, the development, evaluation, and comparison of these techniques in real colonoscopy videos remain largely qualitative due to the difficulty of acquiring ground truth data. In this work, we present a Colonoscopy 3D Video Dataset (C3VD) acquired with a high definition clinical colonoscope and high-fidelity colon models for benchmarking computer vision methods in colonoscopy. We introduce a novel multimodal 2D-3D registration technique to register optical video sequences with ground truth rendered views of a known 3D model. The different modalities are registered by transforming optical images to depth maps with a Generative Adversarial Network and aligning edge features with an evolutionary optimizer. This registration method achieves an average translation error of 0.321 millimeters and an average rotation error of 0.159 degrees in simulation experiments where error-free ground truth is available. The method also leverages video information, improving registration accuracy by 55.6% for translation and 60.4% for rotation compared to single frame registration. 22 short video sequences were registered to generate 10,015 total frames with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models. The dataset and registration source code are available at durr.jhu.edu/C3VD.

Motivation & Objective

• To address the lack of ground truth data for evaluating 3D computer vision techniques in colonoscopy, such as depth estimation and surface reconstruction.
• To develop a robust 2D-3D registration method that enables accurate alignment of real colonoscopy video with synthetic 3D models of the colon.
• To create a benchmark dataset with pixel-level ground truth for depth, surface normals, optical flow, occlusion, and 6-DOF pose to support SLAM, coverage mapping, and lesion detection research.
• To overcome limitations of synthetic renderings by using real clinical colonoscope footage, preserving realistic optics, lighting, and motion.
• To provide open-source 3D models, fabrication protocols, and registration code to enable reproducibility and extension by the research community.

Proposed method

• A novel multimodal 2D-3D registration technique is proposed, combining a Generative Adversarial Network (GAN) to convert optical video frames into depth maps.
• Edge features extracted from optical images and rendered depth maps are aligned using an evolutionary optimizer to estimate 6-DOF camera pose.
• The method leverages temporal consistency by incorporating video sequences, improving registration accuracy by 55.6% for translation and 60.4% for rotation over single-frame registration.
• High-fidelity 3D colon models are created using silicone phantoms with tubular geometry to mimic real colonic anatomy.
• Ground truth data, including depth, surface normals, optical flow, occlusion, and coverage maps, are generated via rendering from the known 3D models.
• The dataset is acquired using a clinical HD colonoscope mounted on a robotic arm to ensure precise trajectory tracking and pose measurement.

Experimental results

Research questions

• RQ1 Can a 2D-3D registration pipeline achieve sub-millimeter and sub-degree accuracy in aligning real colonoscopy video with synthetic 3D models?
• RQ2 To what extent does temporal video information improve the accuracy of 2D-3D registration compared to single-frame methods?
• RQ3 Can a GAN-based depth prediction model effectively bridge the domain gap between real optical images and synthetic depth maps for colonoscopy?
• RQ4 How well can the resulting dataset support the evaluation of 3D reconstruction, visual odometry, and coverage estimation in colonoscopy?
• RQ5 What is the impact of using real clinical endoscope footage versus synthetic renderings on the validity of benchmarking 3D computer vision algorithms?

Key findings

• The proposed 2D-3D registration method achieves an average translation error of 0.321 mm and an average rotation error of 0.159 degrees in simulation with ground truth.
• Incorporating video information improves registration accuracy by 55.6% for translation and 60.4% for rotation compared to single-frame registration.
• The C3VD dataset comprises 22 short video sequences with 10,015 total frames, each labeled with ground truth depth, surface normals, optical flow, occlusion, 6-DOF pose, coverage maps, and 3D models.
• The dataset is recorded using a real HD clinical colonoscope, preserving realistic optical effects such as non-global illumination and sensor noise.
• The 3D model assets, mold files, and fabrication protocols are open-sourced, enabling researchers to reproduce and extend the phantom models.
• The dataset supports benchmarking of SLAM, coverage estimation, polyp detection, and depth estimation algorithms in a clinically relevant setting.