[Paper Review] ERS: a novel comprehensive endoscopy image dataset for machine learning, compliant with the MST 3.0 specification

Jan Cychnerski, Tomasz Dziubich+1 more|arXiv (Cornell University)|Jan 21, 2022

Colorectal Cancer Screening and Detection4 citations

TL;DR

This paper introduces ERS, a large-scale, multi-label endoscopy image dataset with 115,000 approximately labeled frames and 1.23 million unlabeled frames from flexible and capsule endoscopy, fully compliant with the MST 3.0 terminology standard. The dataset enables high-performance training of deep learning models, achieving up to 94% F-score in binary classification and demonstrating significant video length reduction (up to 80%) when using a MobileNet-based classifier to filter normal frames, accelerating clinical review.

ABSTRACT

The article presents a new multi-label comprehensive image dataset from flexible endoscopy, colonoscopy and capsule endoscopy, named ERS. The collection has been labeled according to the full medical specification of 'Minimum Standard Terminology 3.0' (MST 3.0), describing all possible findings in the gastrointestinal tract (104 possible labels), extended with an additional 19 labels useful in common machine learning applications. The dataset contains around 6000 precisely and 115,000 approximately labeled frames from endoscopy videos, 3600 precise and 22,600 approximate segmentation masks, and 1.23 million unlabeled frames from flexible and capsule endoscopy videos. The labeled data cover almost entirely the MST 3.0 standard. The data came from 1520 videos of 1135 patients. Additionally, this paper proposes and describes four exemplary experiments in gastrointestinal image classification task performed using the created dataset. The obtained results indicate the high usefulness and flexibility of the dataset in training and testing machine learning algorithms in the field of endoscopic data analysis.

Motivation & Objective

• To address the critical lack of publicly available, standardized, and comprehensive endoscopy datasets for machine learning in gastrointestinal diagnostics.
• To create a large-scale, multi-label dataset compliant with the Minimum Standard Terminology 3.0 (MST 3.0) for consistent labeling across diverse GI findings.
• To support the development and benchmarking of machine learning models for automated detection of polyps, bleeding, and other GI tract abnormalities in colonoscopy and capsule endoscopy.
• To evaluate the performance of deep learning models on multi-class and binary classification tasks using the ERS dataset, focusing on real-world clinical utility.
• To demonstrate the practical impact of the dataset by showing significant reduction in video review time through automated filtering of normal frames.

Proposed method

• The ERS dataset comprises 1520 endoscopy videos from 1135 patients, including flexible colonoscopy, gastroscopy, and wireless capsule endoscopy, with 6,000 precisely labeled frames and 115,000 approximately labeled frames.
• All images were annotated using the full 104-label MST 3.0 terminology, extended with 19 additional labels relevant for machine learning applications.
• The dataset includes 3,600 precise and 22,600 approximate segmentation masks for lesion localization, supporting both classification and segmentation tasks.
• Four benchmark experiments were conducted using convolutional neural networks (CNNs), including MobileNet, NASNet-Mobile, and others, on multi-class (10-class), mid-level (5-class), and binary (2-class) classification problems.
• A video filtering algorithm based on a trained MobileNet model was applied to remove frames without anomalies, with sensitivity and specificity parameters tuned to balance detection accuracy and video length reduction.
• Performance was evaluated using F-score, with class imbalance addressed via patient-level image count balancing, though this had minimal impact on model performance.

Experimental results

Research questions

• RQ1 Can a large, standardized, and comprehensive endoscopy dataset compliant with MST 3.0 improve the performance and reproducibility of machine learning models in GI diagnostics?
• RQ2 How do different deep learning architectures (e.g., MobileNet, NASNet-Mobile) perform across varying classification complexity (2-class, 5-class, 10-class) on the ERS dataset?
• RQ3 To what extent can automated filtering of normal frames using a trained classifier reduce the time required for clinical video review in capsule endoscopy?
• RQ4 How does the availability of precise vs. approximate annotations affect model performance in lesion detection tasks?
• RQ5 What is the impact of data imbalance and patient-level data distribution on model generalization and F-score in multi-class classification?

Key findings

• The ERS dataset contains 115,000 approximately labeled frames and 1.23 million unlabeled frames from 1,520 endoscopy videos, covering nearly all MST 3.0 terminology categories.
• The 10-class classification problem achieved an F-score of 86–90% for normal tissue, blurry images, and varicose findings, and 75–78% for most diseases, with lower scores for rare conditions like Crohn’s disease (38% F-score) due to limited training data (84 images).
• The 5-class problem achieved 80–85% F-score for diseases and 92% for healthy tissue, outperforming the 10-class setup due to reduced class complexity and absence of low-resource classes.
• The 2-class problem (abnormal vs. healthy tissue) achieved 88% F-score, with MobileNet showing the highest performance, and the use of imprecise annotations slightly improving results.
• The BLUR detection task achieved the highest F-score of 94%, indicating strong model performance for this specific classification task.
• Applying the trained MobileNet classifier to WCE videos reduced video length by 16% at 99% sensitivity, and by over 80% at lower sensitivity thresholds, significantly accelerating clinical review.