[Paper Review] Quick Annotator: an open-source digital pathology based rapid image annotation tool

Runtian Miao, Róbert Tóth+3 more|arXiv (Cornell University)|Jan 6, 2021

AI in cancer detection5 references4 citations

TL;DR

Quick Annotator is an open-source, web-based tool that accelerates digital pathology image annotation by combining real-time user interaction with active deep learning. As users label regions of interest, the model is iteratively retrained and applied to suggest segmentations, achieving 39x to 102x faster annotation speeds while maintaining F-scores above 0.95 across three histologic structures in whole slide images.

ABSTRACT

Image based biomarker discovery typically requires an accurate segmentation of histologic structures (e.g., cell nuclei, tubules, epithelial regions) in digital pathology Whole Slide Images (WSI). Unfortunately, annotating each structure of interest is laborious and often intractable even in moderately sized cohorts. Here, we present an open-source tool, Quick Annotator (QA), designed to improve annotation efficiency of histologic structures by orders of magnitude. While the user annotates regions of interest (ROI) via an intuitive web interface, a deep learning (DL) model is concurrently optimized using these annotations and applied to the ROI. The user iteratively reviews DL results to either (a) accept accurately annotated regions, or (b) correct erroneously segmented structures to improve subsequent model suggestions, before transitioning to other ROIs. We demonstrate the effectiveness of QA over comparable manual efforts via three use cases. These include annotating (a) 337,386 nuclei in 5 pancreatic WSIs, (b) 5,692 tubules in 10 colorectal WSIs, and (c) 14,187 regions of epithelium in 10 breast WSIs. Efficiency gains in terms of annotations per second of 102x, 9x, and 39x were respectively witnessed while retaining f-scores >.95, suggesting QA may be a valuable tool for efficiently fully annotating WSIs employed in downstream biomarker studies.

Motivation & Objective

• Address the critical bottleneck of time-consuming manual annotation in digital pathology image analysis.
• Reduce the labor and time required to annotate large-scale whole slide images (WSIs) for biomarker discovery.
• Enable efficient, scalable, and accurate segmentation of histologic structures such as nuclei, tubules, and epithelial regions.
• Integrate active learning with interactive user feedback to iteratively improve model predictions during annotation.
• Provide a freely available, open-source solution to democratize access to high-efficiency annotation tools in pathology research.

Proposed method

• Utilizes a web-based interface for intuitive, real-time user annotation of regions of interest (ROIs) in whole slide images.
• Employs a deep learning (DL) model that is concurrently trained and updated using newly annotated data from the user.
• Applies the updated model to predict segmentations in real time, offering suggestions for user review and correction.
• Supports iterative refinement: users accept correct predictions or correct errors, which further improves the model’s future suggestions.
• Uses a feedback loop where user corrections are immediately incorporated into the model’s training process to enhance accuracy over time.
• Optimizes annotation efficiency by reducing redundant manual labeling through intelligent, evolving model predictions.

Experimental results

Research questions

• RQ1 Can an interactive, web-based annotation system significantly reduce the time required to annotate histologic structures in whole slide images compared to manual-only approaches?
• RQ2 To what extent does integrating active learning with real-time user feedback improve annotation efficiency and model accuracy?
• RQ3 Can the system maintain high segmentation performance (e.g., F-score >0.95) while achieving substantial speedups in annotation throughput?
• RQ4 How does the tool perform across diverse histologic structures (e.g., nuclei, tubules, epithelium) in different tissue types?
• RQ5 Is the open-source nature of the tool sufficient to support broad adoption and integration into large-scale biomarker discovery pipelines?

Key findings

• Quick Annotator achieved a 102x speedup in annotating 337,386 nuclei across 5 pancreatic whole slide images compared to manual annotation.
• The tool enabled a 9x speedup in annotating 5,692 tubules across 10 colorectal whole slide images while maintaining an F-score above 0.95.
• For 14,187 epithelial regions in 10 breast whole slide images, the system delivered a 39x improvement in annotation efficiency with F-scores exceeding 0.95.
• The iterative feedback loop between user corrections and model retraining significantly enhanced model accuracy over time.
• The system demonstrated robust performance across diverse tissue types and histologic structures, indicating broad applicability.
• The open-source deployment ensures accessibility and extensibility for integration into large-scale digital pathology research workflows.