[Paper Review] Video Polyp Segmentation: A Deep Learning Perspective

Ge-Peng Ji, Guobao Xiao+5 more|arXiv (Cornell University)|Mar 27, 2022

Colorectal Cancer Screening and Detection4 citations

TL;DR

This paper introduces SUN-SEG, a large-scale video polyp segmentation (VPS) dataset with diverse annotations (mask, boundary, scribble, polygon, attribute), and proposes PNS+, a lightweight deep learning model using global-to-local encoding with normalized self-attention to capture long- and short-term spatiotemporal dependencies. PNS+ achieves state-of-the-art performance with 170 FPS inference speed on the SUN-SEG benchmark, setting a new standard for real-time, accurate VPS.

ABSTRACT

We present the first comprehensive video polyp segmentation (VPS) study in the deep learning era. Over the years, developments in VPS are not moving forward with ease due to the lack of large-scale fine-grained segmentation annotations. To address this issue, we first introduce a high-quality frame-by-frame annotated VPS dataset, named SUN-SEG, which contains 158,690 colonoscopy frames from the well-known SUN-database. We provide additional annotations with diverse types, i.e., attribute, object mask, boundary, scribble, and polygon. Second, we design a simple but efficient baseline, dubbed PNS+, consisting of a global encoder, a local encoder, and normalized self-attention (NS) blocks. The global and local encoders receive an anchor frame and multiple successive frames to extract long-term and short-term spatial-temporal representations, which are then progressively updated by two NS blocks. Extensive experiments show that PNS+ achieves the best performance and real-time inference speed (170fps), making it a promising solution for the VPS task. Third, we extensively evaluate 13 representative polyp/object segmentation models on our SUN-SEG dataset and provide attribute-based comparisons. Finally, we discuss several open issues and suggest possible research directions for the VPS community.

Motivation & Objective

• To address the lack of large-scale, finely-annotated video polyp segmentation datasets by creating SUN-SEG with 158,690 frame-level annotations.
• To develop a robust, real-time VPS baseline model that effectively captures both long-term and short-term spatiotemporal dependencies.
• To establish the first comprehensive benchmark evaluating 13 state-of-the-art polyp and object segmentation models on the new SUN-SEG dataset.
• To identify persistent challenges in VPS under real-world colonoscopy conditions and suggest future research directions.

Proposed method

• Proposes a global-to-local learning paradigm: a global encoder processes the anchor frame, while local encoders process successive frames to extract long- and short-term spatiotemporal features.
• Introduces normalized self-attention (NS) blocks that dynamically refine feature representations by attending to spatial and temporal cues across features.
• Uses a multi-stream architecture where the global and local encoders are fused before processing through two NS blocks for progressive refinement.
• Employs a simple yet effective architecture with shared backbones and lightweight attention modules to ensure real-time inference (170 FPS).
• Leverages five annotation types—attribute, object mask, boundary, scribble, and polygon—on the SUN-SEG dataset to support diverse downstream tasks.
• Conducts extensive ablation and comparative studies across 13 models to evaluate performance under various challenging attributes (e.g., HO, LO, SV, FM, OV).

Experimental results

Research questions

• RQ1 How can a large-scale, multi-annotated video polyp segmentation dataset be constructed to support diverse medical AI tasks?
• RQ2 What is the optimal architecture design for real-time, accurate video polyp segmentation that models both long- and short-term temporal dependencies?
• RQ3 How do existing state-of-the-art polyp segmentation models perform under challenging colonoscopy conditions such as occlusion, low contrast, and specular reflections?
• RQ4 What are the key failure modes of current models in VPS, and how can they be systematically diagnosed and addressed?
• RQ5 What future research directions are most promising for advancing robust, trustworthy, and privacy-preserving VPS in clinical settings?

Key findings

• PNS+ achieves the highest performance on the SUN-SEG dataset, outperforming 13 competing models across multiple evaluation metrics.
• The model achieves real-time inference speed of 170 FPS, making it suitable for clinical deployment in endoscopy systems.
• Models consistently fail on hard attributes such as HO (highly irregular shape), LO (low contrast), and SV (small size), indicating a need for better feature learning.
• False positives and negatives occur on surgical instruments and optical flares, highlighting limitations in semantic understanding and polyp-specific representation learning.
• Lack of temporal modeling leads to performance degradation in occlusion cases (e.g., OV, OC), suggesting that better video-level reasoning is needed.
• The benchmark reveals that current SOTA models still have sensitivity scores below 0.63 on the most challenging SUN-SEG-Hard subset, indicating significant room for improvement.