[Paper Review] A multi-centre polyp detection and segmentation dataset for generalisability assessment

Sharib Ali, Debesh Jha+13 more|arXiv (Cornell University)|Jun 8, 2021

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This paper introduces PolypGen, a large-scale, multi-centre colonoscopy dataset with 3,762 polyp annotations across 300+ patients, including single frames and video sequences. It enables rigorous evaluation of polyp detection and segmentation models, demonstrating that residual networks with atrous spatial pyramid pooling improve generalization across diverse clinical settings, with ResNetUNet achieving real-time inference (87 FPS) and superior boundary accuracy (MASD: 27.10 on sequences).

ABSTRACT

Polyps in the colon are widely known cancer precursors identified by colonoscopy. Whilst most polyps are benign, the polyp's number, size and surface structure are linked to the risk of colon cancer. Several methods have been developed to automate polyp detection and segmentation. However, the main issue is that they are not tested rigorously on a large multicentre purpose-built dataset, one reason being the lack of a comprehensive public dataset. As a result, the developed methods may not generalise to different population datasets. To this extent, we have curated a dataset from six unique centres incorporating more than 300 patients. The dataset includes both single frame and sequence data with 3762 annotated polyp labels with precise delineation of polyp boundaries verified by six senior gastroenterologists. To our knowledge, this is the most comprehensive detection and pixel-level segmentation dataset (referred to as extit{PolypGen}) curated by a team of computational scientists and expert gastroenterologists. The paper provides insight into data construction and annotation strategies, quality assurance, and technical validation. Our dataset can be downloaded from \url{ https://doi.org/10.7303/syn26376615}.

Motivation & Objective

• To address the lack of comprehensive, multi-centre datasets for polyp detection and segmentation in colonoscopy.
• To improve model generalization across diverse clinical populations and imaging conditions by providing a large-scale, multi-centre dataset with expert-verified annotations.
• To enable benchmarking of deep learning models on both single-frame and sequential colonoscopy data, supporting evaluation of temporal modeling and domain shift robustness.
• To support the development of federated learning and transfer learning approaches by providing centre-specific data splits.
• To reduce inter-observer variability in polyp assessment by providing a standardized, high-quality dataset for training and validating AI systems.

Proposed method

• Curated colonoscopy data from six distinct medical centres, including both single-frame images and endoscopic video sequences.
• Generated 3,762 precise pixel-level polyp segmentation masks through consensus annotation by six senior gastroenterologists.
• Implemented a multi-stage data curation pipeline involving quality control, data balancing, and expert review to ensure annotation reliability.
• Provided structured data splits for centre-based evaluation, enabling assessment of model generalization across different clinical sites.
• Developed benchmark code and evaluation tools for segmentation performance, including metrics like DSC, NSD, MASD, and HD for boundary accuracy.
• Utilized state-of-the-art deep learning architectures (e.g., ResNetUNet, DeepLabV3+) for technical validation and performance benchmarking on the dataset.

Experimental results

Research questions

• RQ1 How do existing deep learning models for polyp segmentation perform on a diverse, multi-centre dataset with real-world imaging variability?
• RQ2 To what extent does model performance degrade when evaluated on data from centres not seen during training, and how can this be mitigated?
• RQ3 Which network architectures and components (e.g., residual connections, atrous spatial pyramid pooling) best improve generalization across different clinical settings?
• RQ4 How do boundary-based metrics like MASD compare to standard metrics like DSC in evaluating polyp segmentation performance, especially for small and medium-sized polyps?
• RQ5 Can the proposed dataset support the development and evaluation of federated learning and transfer learning strategies for colonoscopy AI systems?

Key findings

• ResNetUNet with a ResNet34 backbone achieved real-time inference at 87 FPS while maintaining high segmentation accuracy (DSC comparable to DeepLabV3+ with ResNet50).
• The MASD metric revealed that ResNetUNet outperformed DeepLabV3+ in boundary accuracy, particularly on small and medium polyps, with scores of 35.83 (single) and 27.10 (sequence) vs. 41.04 and 28.19, respectively.
• Baseline architectures like vanilla UNet, PSPNet, and FCN8 showed sub-optimal performance due to limited capacity to handle polyp size variation and image contrast changes.
• Models trained on combined multi-centre data had higher false positive rates, while centre-specific validation splits revealed significant performance variation, highlighting the importance of domain-aware evaluation.
• The best-performing models detected polyps clearly visible with high contrast or distinct mucosal elevation, while failing on flat, sessile, or low-contrast lesions.
• The dataset's diverse data distribution, including varying illumination, view angles, and polyp morphology, exposed limitations in current models, especially for small polyps (<100×100 pixels).