[Paper Review] Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery

Sebastian Bodenstedt, Max Allan+18 more|arXiv (Cornell University)|May 7, 2018

Surgical Simulation and TrainingMedicine20 references50 citations

TL;DR

The paper provides a comprehensive, data-set–driven comparison of vision-based instrument segmentation and tracking methods for robotic and conventional laparoscopic surgery, showing deep learning methods outperform traditional approaches for segmentation and that merging multiple methods improves accuracy; tracking remains challenging.

ABSTRACT

Intraoperative segmentation and tracking of minimally invasive instruments is a prerequisite for computer- and robotic-assisted surgery. Since additional hardware like tracking systems or the robot encoders are cumbersome and lack accuracy, surgical vision is evolving as promising techniques to segment and track the instruments using only the endoscopic images. However, what is missing so far are common image data sets for consistent evaluation and benchmarking of algorithms against each other. The paper presents a comparative validation study of different vision-based methods for instrument segmentation and tracking in the context of robotic as well as conventional laparoscopic surgery. The contribution of the paper is twofold: we introduce a comprehensive validation data set that was provided to the study participants and present the results of the comparative validation study. Based on the results of the validation study, we arrive at the conclusion that modern deep learning approaches outperform other methods in instrument segmentation tasks, but the results are still not perfect. Furthermore, we show that merging results from different methods actually significantly increases accuracy in comparison to the best stand-alone method. On the other hand, the results of the instrument tracking task show that this is still an open challenge, especially during challenging scenarios in conventional laparoscopic surgery.

Motivation & Objective

• Provide a common, publicly available validation data set for instrument segmentation and tracking in robotic and conventional laparoscopy.
• Compare state-of-the-art vision-based segmentation and tracking methods on standardized robotic and conventional laparoscopic data.
• Assess whether combining methods improves segmentation accuracy beyond best standalone methods.
• Identify remaining challenges in instrument tracking under realistic surgical conditions.

Proposed method

• Curate and release two validation data sets: robotic (articulated instruments) and conventional laparoscopic (rigid instruments), with training/testing splits and annotated masks.
• Evaluate multiple segmentation approaches (CNN-based and RF-based) from different groups on the two data sets.
• Evaluate multiple tracking approaches that extend segmentation results with motion estimation and pose tracking.
• Investigate merging segmentation outputs from multiple methods using majority voting and STAPLE to improve accuracy.
• Use Dice similarity coefficient (DSC) as the primary metric, with additional measures like precision, recall, and accuracy; statistical significance via Wilcoxon signed-rank tests.

Experimental results

Research questions

• RQ1 How do current vision-based instrument segmentation methods perform on robotic versus conventional laparoscopic data with realistic challenges (occlusion, smoke, bleeding, meshes)?
• RQ2 Can merging multiple segmentation results improve Dice similarity coefficient beyond the best single method across data sets and challenging scenarios?
• RQ3 Which tracking approaches best estimate instrument center, axis, and orientation under different surgical settings, and what are the main sources of error?
• RQ4 What are the remaining limitations of instrument tracking in conventional laparoscopic surgery under challenging conditions?

Key findings

• CNN-based segmentation methods generally outperform non-CNN methods on both robotic and conventional datasets.
• SEG-KIT-CNN achieves top performance on D-CONV-SEG All with DSC 0.88; SEG-JHU leads on D-ROB-SEG with DSC 0.88.
• Merging segmentation results via majority voting or STAPLE can significantly surpass the best single method, with similar gains across datasets.
• For D-CONV-SEG All, the top three merged configurations exceed the best single method, with SEG-KIT-CNN frequently included among top merges.
• Instrument tracking remains challenging in conventional laparoscopic surgery, with performance depending on occlusions, bleeding, smoke, and mesh presence.
• Merged tracking outputs from multiple methods also improve tracking accuracy, though gains depend on scenario and method complementarity.