[Paper Review] 2017 Robotic Instrument Segmentation Challenge

Max Allan, Alexey A. Shvets+17 more|arXiv (Cornell University)|Feb 18, 2019

Anatomy and Medical Technology29 references56 citations

TL;DR

A MICCAI 2017 challenge presenting hand-labeled, multi-task segmentation of da Vinci robotic instruments across 10 datasets, with binary, parts, and type segmentation tasks and multiple participating methods.

ABSTRACT

In mainstream computer vision and machine learning, public datasets such as ImageNet, COCO and KITTI have helped drive enormous improvements by enabling researchers to understand the strengths and limitations of different algorithms via performance comparison. However, this type of approach has had limited translation to problems in robotic assisted surgery as this field has never established the same level of common datasets and benchmarking methods. In 2015 a sub-challenge was introduced at the EndoVis workshop where a set of robotic images were provided with automatically generated annotations from robot forward kinematics. However, there were issues with this dataset due to the limited background variation, lack of complex motion and inaccuracies in the annotation. In this work we present the results of the 2017 challenge on robotic instrument segmentation which involved 10 teams participating in binary, parts and type based segmentation of articulated da Vinci robotic instruments.

Motivation & Objective

• Motivate scene understanding in robotic minimally invasive surgery through instrument segmentation.
• Provide a high-quality, diverse dataset with manual labels to enable fair method comparison.
• Evaluate binary, parts-based, and type-based segmentation across multiple instruments and procedures.
• Encourage development of real-time or near-real-time segmentation methods suitable for guidance and AR overlays.

Proposed method

• Three sub-tasks: binary segmentation of instruments vs background, parts segmentation (shaft, wrist, jaws), and type-based segmentation (instrument types).
• Dataset collected from 10 porcine abdominal procedures with 300 frames per sequence; training on first 225 frames and test on last 75 frames per sequence; stereo left-eye images provided.
• Ground truth labels created manually by Intuitive Surgical segmentation team with frame-by-frame polygons.
• 11 participating teams contributed various CNN-based approaches including FCN/UNet variants, TernausNet, SegNet, and handcrafted baselines.
• Evaluation based on mean IoU across present classes per frame, with dataset-weighted overall scores.
• Several teams used pre-trained encoders (VGG, ResNet variants) and multi-scale or cascaded architectures to improve accuracy.

Experimental results

Research questions

• RQ1 How well can state-of-the-art segmentation architectures perform on binary instrument segmentation in robotic surgery images?
• RQ2 Can multi-task segmentation (binary, parts, and type) be effectively learned from manually labeled, diverse surgical data?
• RQ3 What is the impact of data diversity (8 training sequences, 10 test sequences) on generalization across datasets?
• RQ4 How do different network architectures and training strategies compare in terms of mean IoU across tasks and datasets?

Key findings

• MIT achieved the highest average mean IoU in binary segmentation (0.854) across the 10 datasets.
• Average mean IoU for binary segmentation across all datasets and teams ranged with MIT 0.888, UB 0.875, and others varying (see Table I).
• For parts segmentation, MIT achieved the highest mean IoU on 7 of 10 datasets, with overall mean IoU 0.737 for MIT (Table II).
• In type-based segmentation, multi-team participation was lower and overall mean IoU varied (Table VI).
• Overall, deep learning architectures with pre-trained encoders and multi-scale or cascaded designs performed best, while simple baselines and non-ML methods were competitive on some frames (e.g., UW baseline).
• The challenge emphasized the need for high-quality, manually labeled data and diverse background variation to drive advancement in robotic instrument segmentation.