[Paper Review] Endoscopy Disease Detection and Segmentation (EDD2020)

Sharib Ali, Barbara Braden+8 more|arXiv (Cornell University)|Jan 15, 2020

Gastrointestinal Bleeding Diagnosis and Treatment3 references15 citations

TL;DR

This paper introduces the EDD2020 challenge, a multi-center, multi-organ, multi-modality dataset of 386 annotated gastrointestinal endoscopy frames for disease detection and segmentation. It evaluates deep learning models on bounding box localization, pixel-level segmentation, and out-of-sample generalization using mAP, IoU, and F1-score metrics, with a top ensemble model achieving 45% average mAP despite class imbalance and challenging cases.

ABSTRACT

Endoscopy is a widely used clinical procedure for the early detection of cancers in hollow-organs such as oesophagus, stomach, and colon. Computer-assisted methods for accurate and temporally consistent localisation and segmentation of diseased region-of-interests enable precise quantification and mapping of lesions from clinical endoscopy videos which is critical for monitoring and surgical planning. Innovations have the potential to improve current medical practices and refine healthcare systems worldwide. However, well-annotated, representative publically-available datasets for disease detection for assessing reproducibility and facilitating standardised comparison of methods is still lacking. Many methods to detect diseased regions in endoscopy have been proposed however these have primarily focussed on the task of polyp detection in the gastrointestinal tract with demonstration on datasets acquired from at most a few data centres and single modality imaging, most commonly white light. Here, we present our multi-class disease detection and segmentation challenge in clinical endoscopy. With this sub-challenge we aim to establish a comprehensive dataset to benchmark algorithms for disease detection.

Motivation & Objective

• To address the lack of comprehensive, publicly available, multi-center datasets for endoscopy disease detection and segmentation.
• To establish a benchmark for evaluating deep learning models on real-world clinical data across multiple organs and imaging modalities.
• To promote research on robust, generalizable, and clinically applicable computer-aided diagnosis systems in endoscopy.
• To evaluate model performance on detection, segmentation, and out-of-sample generalization using standardized metrics.
• To facilitate reproducible research and comparative evaluation of algorithms through a crowd-sourced challenge framework.

Proposed method

• The dataset comprises 386 endoscopy frames from four international centers, covering colon, esophagus, and stomach with five disease classes: NDBE, suspicious lesion, high-grade dysplasia, cancer, and polyp.
• Annotations were performed by clinical experts and post-doctoral researchers using the VIA tool, with bounding boxes in PASCAL VOC format and segmentation masks as 5-channel TIFF images.
• Evaluation includes three tasks: detection (bounding box localization), semantic segmentation (pixel-level labeling), and out-of-sample detection (generalization to unseen institutions).
• Performance is measured using mAPd (mean average precision across IoU thresholds 0.25–0.75 in steps of 0.05) and IoU for detection, and F1, F2, precision, and recall for segmentation.
• A weighted score (0.6×mAPd + 0.4×IoUd) ranks participants, with standard deviation used for tie-breaking.
• Out-of-sample generalization is evaluated via the deviation score |mAPd − mAPg|, where lower values indicate better robustness.

Experimental results

Research questions

• RQ1 How well do deep learning models perform in detecting and localizing multiple gastrointestinal diseases across diverse organs and imaging modalities?
• RQ2 To what extent can models generalize to unseen institutions and data distributions not included in training or validation?
• RQ3 How do mAP and IoU metrics correlate with clinical relevance in endoscopy image analysis?
• RQ4 What impact does class imbalance (e.g., few cancer cases) have on model performance and generalization?
• RQ5 Can ensemble methods improve detection and segmentation performance on challenging, ambiguous, or overlapping lesions?

Key findings

• The overall average mAP across the entire test dataset was 35%, primarily due to insufficient cancer samples and complex cases like polyps overlapping with Barrett’s tissue.
• A baseline algorithm achieved 58.52% mAP on a subset of 8 test images, with mAP25 and mAP50 both at 62.5%, indicating strong localization on representative cases.
• An ensemble of three models improved the average mAP to over 45%, demonstrating the benefit of model combination in handling ambiguous or rare cases.
• Algorithm 1 failed to detect cancer and misclassified small Barrett’s islands as polyps, highlighting challenges in subtle lesion recognition.
• The best-performing models achieved low deviation scores (devg → 0), indicating strong out-of-sample generalization with consistent mAP across training and unseen data.
• The challenge revealed that high mAP on small subsets does not guarantee strong performance on the full dataset, underscoring the importance of robustness and generalization.