[Paper Review] Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos

Abdullah Hamdi, Changchun Yang+1 more|arXiv (Cornell University)|Mar 26, 2026

Colorectal Cancer Screening and Detection0 citations

TL;DR

Colon-Bench introduces a multi-stage agentic workflow to densely annotate full-procedure colonoscopy videos, creating a large, multi-task benchmark for evaluating state-of-the-art Multimodal Large Language Models on lesion detection, segmentation, and VQA. It also analyzes MLLM performance and proposes a prompting strategy to boost zero-shot results.

ABSTRACT

Early screening via colonoscopy is critical for colon cancer prevention, yet developing robust AI systems for this domain is hindered by the lack of densely annotated, long-sequence video datasets. Existing datasets predominantly focus on single-class polyp detection and lack the rich spatial, temporal, and linguistic annotations required to evaluate modern Multimodal Large Language Models (MLLMs). To address this critical gap, we introduce Colon-Bench, generated via a novel multi-stage agentic workflow. Our pipeline seamlessly integrates temporal proposals, bounding-box tracking, AI-driven visual confirmation, and human-in-the-loop review to scalably annotate full-procedure videos. The resulting verified benchmark is unprecedented in scope, encompassing 528 videos, 14 distinct lesion categories (including polyps, ulcers, and bleeding), over 300,000 bounding boxes, 213,000 segmentation masks, and 133,000 words of clinical descriptions. We utilize Colon-Bench to rigorously evaluate state-of-the-art MLLMs across lesion classification, Open-Vocabulary Video Object Segmentation (OV-VOS), and video Visual Question Answering (VQA). The MLLM results demonstrate surprisingly high localization performance in medical domains compared to SAM-3. Finally, we analyze common VQA errors from MLLMs to introduce a novel "colon-skill" prompting strategy, improving zero-shot MLLM performance by up to 9.7% across most MLLMs. The dataset and the code are available at https://abdullahamdi.com/colon-bench .

Motivation & Objective

• Address the lack of densely annotated, long-sequence colonoscopy video datasets.
• Create a scalable annotation pipeline combining vision-language models, tracking, AI confirmation, and human review.
• Provide a comprehensive benchmark for lesion detection, segmentation, and language-based understanding in colonoscopy.
• Enable evaluation of Open-Vocabulary Video Segmentation (OV-VOS) and VQA in medical video contexts.

Proposed method

• Develop a novel multi-stage agentic workflow (proposal generation, verification, cued AI confirmation, and human review) to yield high-quality dense annotations.
• Use EdgeTAM/SAM-based tracking to propagate spatial annotations across frames.
• Incorporate AI-driven confirmation overlays and clinician review as a final quality gate.
• Assemble Colon-Bench with 14 lesion categories, bounding boxes, segmentation masks, and 133k words of clinical text.
• Evaluate multiple state-of-the-art MLLMs on four tasks: binary lesion classification, OV-VOS, and two VQA difficulties.
• Introduce a Colon-Skill prompting strategy derived from cross-model error analysis to improve zero-shot VQA performance.

Figure 1: Colon-Bench Annotation Pipeline. Overview of the multi-stage agentic workflow used to build the dataset, from VLM proposal generation through verification, tracking with spatial annotations, AI confirmation, and clinician review. The figure highlights how successive filters reduce false po

Experimental results

Research questions

• RQ1 How effectively can an agentic workflow produce dense, high-quality colonoscopy annotations at scale?
• RQ2 How well do current MLLMs perform on colonoscopy-specific tasks (classification, OV-VOS, VQA) using Colon-Bench?
• RQ3 Does structured domain knowledge prompting improve MLLM performance on medical VQA in colonoscopy datasets?
• RQ4 What is the impact of temporal context on segmentation and VQA performance in long-form colonoscopy videos.

Key findings

• The Colon-Bench pipeline annotates 528 windows across 464,035 frames, yielding 300,132 bounding boxes, 213,067 segmentation masks, and 133,289 words of clinical text.
• Colon-Bench provides four evaluation tasks: binary classification, OV-VOS, and two VQA difficulties (prompted and unprompted).
• Open-weight MLLMs like Seed 1.6 and Gemini 3 variants achieve strong performance across tasks, with Gemini 3 Pro/Flash leading VQA and segmentation metrics.
• Open-vocabulary segmentation with GPT-5.4 + EdgeTAM beats SAM-3 by 32.0% mIoU in video segmentation.
• A novel Colon-Skill prompting strategy improves zero-shot MLLM VQA performance by up to 9.7% across most models.
• Ablation shows that using three-frame detections provides a good balance between segmentation quality and computation.

Figure 2: Lesion Category Distribution. Long-tailed lesion category distribution in Colon-Bench, highlighting the diversity of lesions.