[Paper Review] m2caiSeg: Semantic Segmentation of Laparoscopic Images using Convolutional Neural Networks

Salman Maqbool, Aqsa Riaz+2 more|arXiv (Cornell University)|Aug 23, 2020

Surgical Simulation and Training34 references18 citations

TL;DR

This paper proposes m2caiSeg, a novel dataset and deep learning framework for semantic segmentation of laparoscopic surgical images using a U-Net-based encoder-decoder CNN with self-supervised pre-training and data augmentation. The method achieves a F1 score of 0.57 when classifying organs and instruments collectively, demonstrating its potential as a foundational step toward autonomous robotic surgery.

ABSTRACT

Autonomous surgical procedures, in particular minimal invasive surgeries, are the next frontier for Artificial Intelligence research. However, the existing challenges include precise identification of the human anatomy and the surgical settings, and modeling the environment for training of an autonomous agent. To address the identification of human anatomy and the surgical settings, we propose a deep learning based semantic segmentation algorithm to identify and label the tissues and organs in the endoscopic video feed of the human torso region. We present an annotated dataset, m2caiSeg, created from endoscopic video feeds of real-world surgical procedures. Overall, the data consists of 307 images, each of which is annotated for the organs and different surgical instruments present in the scene. We propose and train a deep convolutional neural network for the semantic segmentation task. To cater for the low quantity of annotated data, we use unsupervised pre-training and data augmentation. The trained model is evaluated on an independent test set of the proposed dataset. We obtained a F1 score of 0.33 while using all the labeled categories for the semantic segmentation task. Secondly, we labeled all instruments into an 'Instruments' superclass to evaluate the model's performance on discerning the various organs and obtained a F1 score of 0.57. We propose a new dataset and a deep learning method for pixel level identification of various organs and instruments in a endoscopic surgical scene. Surgical scene understanding is one of the first steps towards automating surgical procedures.

Motivation & Objective

• To address the critical challenge of surgical scene understanding in autonomous robotic surgery by enabling pixel-level identification of organs and instruments.
• To create a new, publicly available dataset, m2caiSeg, with 307 annotated laparoscopic images for training and evaluating semantic segmentation models.
• To develop a deep learning-based semantic segmentation framework that generalizes well despite limited annotated data, using unsupervised pre-training and data augmentation.
• To establish a baseline performance for future research in laparoscopic scene segmentation, particularly for safety-critical applications.
• To promote reproducibility and community advancement by releasing the dataset and code openly.

Proposed method

• Proposed a U-Net-style encoder-decoder convolutional neural network architecture to perform pixel-level segmentation of organs and instruments in laparoscopic images.
• Applied self-supervised pre-training using a reconstruction loss on the ImageNet dataset to learn generalizable features before fine-tuning on the m2caiSeg dataset.
• Employed extensive data augmentation techniques, including random cropping, flipping, and color jittering, to mitigate data scarcity and improve model robustness.
• Fine-tuned the pre-trained model on the m2caiSeg dataset using cross-entropy loss and Adam optimization with a cosine annealing learning rate schedule.
• Evaluated the model on a held-out test set using standard segmentation metrics: Intersection over Union (IoU), precision, recall, and F1 score.
• Conducted transfer learning experiments by fine-tuning the m2caiSeg model on the EndoVis 2018 Robotic Scene Segmentation dataset to assess generalization capability.

Experimental results

Research questions

• RQ1 Can self-supervised pre-training improve semantic segmentation performance in low-data regimes typical of surgical imaging?
• RQ2 How well can a deep learning model generalize to unseen surgical scenes when trained on a limited, yet diverse, dataset of laparoscopic images?
• RQ3 What is the performance of a segmentation model when distinguishing between fine-grained structures like surgical instruments and small anatomical features such as needles and threads?
• RQ4 To what extent does the proposed architecture preserve spatial details and boundaries in complex, cluttered surgical scenes?
• RQ5 Can the model trained on m2caiSeg generalize to other surgical datasets, such as EndoVis 2018, with minimal fine-tuning?

Key findings

• The model achieved a F1 score of 0.33 when classifying all 10 categories of organs and instruments in the m2caiSeg dataset, indicating moderate performance on fine-grained segmentation.
• When instruments were grouped into a single 'Instruments' superclass, the F1 score improved to 0.57, demonstrating better generalization on broader categories.
• The model achieved high IoU (0.876) and F1 (0.934) for the background and tissue classes on the EndoVis 2018 dataset, indicating strong performance on dominant and larger structures.
• Performance dropped significantly on small or rare objects such as needles, threads, and clamps, with F1 scores of 0.000, highlighting limitations in detecting fine details.
• The model generalized reasonably well to the EndoVis 2018 dataset after fine-tuning, achieving F1 scores above 0.80 for major classes like instrument shaft and covered kidney.
• The results suggest that self-supervised pre-training followed by fine-tuning is effective for surgical scene segmentation, even with limited annotated data.