[Paper Review] Depth Reconstruction and Computer-Aided Polyp Detection in Optical Colonoscopy Video Frames

Saad Nadeem, Arie Kaufman|arXiv (Cornell University)|Sep 5, 2016

Advanced Image and Video Retrieval Techniques22 references11 citations

TL;DR

This paper proposes a computer-aided detection system for polyps in optical colonoscopy using depth map estimation from virtual colonoscopy (VC) training data. By leveraging a machine learning-based RGB-depth dictionary trained on VC datasets, the method infers depth in 2D optical colonoscopy frames without requiring camera parameters or geometric assumptions, achieving 84.0% recall and 83.4% specificity in polyp detection.

ABSTRACT

We present a computer-aided detection algorithm for polyps in optical colonoscopy images. Polyps are the precursors to colon cancer. In the US alone, more than 14 million optical colonoscopies are performed every year, mostly to screen for polyps. Optical colonoscopy has been shown to have an approximately 25% polyp miss rate due to the convoluted folds and bends present in the colon. In this work, we present an automatic detection algorithm to detect these polyps in the optical colonoscopy images. We use a machine learning algorithm to infer a depth map for a given optical colonoscopy image and then use a detailed pre-built polyp profile to detect and delineate the boundaries of polyps in this given image. We have achieved the best recall of 84.0% and the best specificity value of 83.4%.

Motivation & Objective

• Address the high polyp miss rate (~25%) in optical colonoscopy due to complex colon anatomy and limited 3D information.
• Overcome limitations of traditional computer vision techniques (e.g., shape-from-stereo, shape-from-shading) in monocular, dynamic colonoscopy environments.
• Enable accurate polyp detection in 2D optical colonoscopy by estimating 3D depth features using machine learning and VC-derived training data.
• Develop a robust, camera-parameter-free method for depth reconstruction and polyp boundary delineation in endoscopic video frames.
• Extend the framework to other endoscopic procedures (e.g., bronchoscopy) by leveraging generalizable depth representation.

Proposed method

• Construct an RGB-depth dictionary using virtual colonoscopy (VC) datasets, where depth maps are derived from 3D CT-based colon reconstructions.
• Apply a machine learning-based mapping from input optical colonoscopy (OC) RGB images to corresponding depth maps using the pre-trained RGB-depth dictionary.
• Use a pre-built database of polyp profiles in depth map space to detect and delineate polyp boundaries based on 3D geometric features.
• Leverage structural similarity and normalized error metrics (NRMSE, HD, SSIM) to evaluate depth map reconstruction quality.
• Apply the method to real OC video frames without assuming intrinsic camera parameters or rigid colon geometry.
• Use a nearest-neighbor-like matching strategy to transfer depth representations from VC to OC images, enabling dense depth estimation.

Experimental results

Research questions

• RQ1 Can depth maps be effectively inferred from 2D optical colonoscopy images using a machine learning model trained on virtual colonoscopy data?
• RQ2 To what extent does incorporating 3D geometric depth information improve polyp detection performance in 2D colonoscopy frames?
• RQ3 How robust is the proposed method to variations in viewing angle, texture, and lack of camera calibration in endoscopic imaging?
• RQ4 Can the depth estimation framework be generalized to other endoscopic procedures beyond colonoscopy?
• RQ5 What is the performance of the system in terms of recall and specificity for polyp detection in real clinical video frames?

Key findings

• The proposed method achieved a best recall of 84.0% and a specificity of 83.4% in detecting polyps across 300 endoscopy video frames, with 75 polyp and 225 normal frames.
• Depth map reconstruction using the RGB-depth dictionary yielded NRMSE of 0.43, HD of 0.43, and SSIM of 0.30 under optimal texture conditions.
• The method successfully reconstructed dense depth maps from OC images without requiring intrinsic camera parameters or assumptions about colon rigidity.
• Failure cases were primarily due to texture similarity between polyps and background, as shown in Figure 9, where polyps were not detected despite correct depth estimation.
• The framework demonstrated generalization potential, with successful depth estimation applied to bronchoscopy video frames using the same dictionary.
• The approach outperformed prior methods such as Hong et al. [15], which produced distorted reconstructions and could not handle surface views, as shown in Figure 8.