[Paper Review] Segmentation of Surgical Instruments for Minimally-Invasive Robot-Assisted Procedures Using Generative Deep Neural Networks

Iñigo Azqueta-Gavaldon, Florian A. Fröhlich+2 more|elib (German Aerospace Center)|Jun 5, 2020

Surgical Simulation and Training16 references4 citations

TL;DR

This paper proposes a two-stage method that uses CycleGAN-based domain adaptation to generate realistic synthetic endoscopic images of MiroSurge surgical instruments, enabling high-quality semantic segmentation via a U-Net model trained on these augmented data. The approach achieves 88.06% mean Intersection over Union (IoU) on real endoscopic images and shows promising zero-shot generalization to unseen da Vinci instruments despite domain shift.

ABSTRACT

This work proves that semantic segmentation on minimally invasive surgical instruments can be improved by using training data that has been augmented through domain adaptation. The benefit of this method is twofold. Firstly, it suppresses the need of manually labeling thousands of images by transforming synthetic data into realistic-looking data. To achieve this, a CycleGAN model is used, which transforms a source dataset to approximate the domain distribution of a target dataset. Secondly, this newly generated data with perfect labels is utilized to train a semantic segmentation neural network, U-Net. This method shows generalization capabilities on data with variability regarding its rotation- position- and lighting conditions. Nevertheless, one of the caveats of this approach is that the model is unable to generalize well to other surgical instruments with a different shape from the one used for training. This is driven by the lack of a high variance in the geometric distribution of the training data. Future work will focus on making the model more scale-invariant and able to adapt to other types of surgical instruments previously unseen by the training.

Motivation & Objective

• To reduce the cost and effort of manual annotation for semantic segmentation in robot-assisted minimally invasive surgery.
• To address the domain gap between synthetic training data and real endoscopic images, which hinders model generalization.
• To improve segmentation performance on real surgical instrument images using only synthetic data augmented via domain adaptation.
• To evaluate the generalization capability of the trained model on a different surgical system (da Vinci) with distinct instrument geometry and appearance.

Proposed method

• A CycleGAN is trained on unpaired synthetic and real endoscopic images to translate synthetic instrument renderings into realistic-looking endoscopic appearances.
• The domain-adapted synthetic images are used to train a U-Net-based semantic segmentation network with heavy data augmentation.
• The training data are carefully aligned in pose and geometry distribution between synthetic and real domains to improve translation fidelity.
• The U-Net model is trained end-to-end on the generated, realistic-looking synthetic dataset with perfect ground-truth segmentation masks.
• Model generalization is evaluated on real MiroSurge images and on a held-out da Vinci instrument dataset with diverse instrument types and multiple instruments per image.
• The method leverages the strong feature extraction and scale-aware design of U-Net, with potential for future integration of MultiResUNet or attention mechanisms.

Experimental results

Research questions

• RQ1 Can domain adaptation via CycleGAN effectively bridge the domain gap between synthetic and real endoscopic images of surgical instruments?
• RQ2 Does training a semantic segmentation model on domain-adapted synthetic data yield superior performance on real surgical instrument segmentation compared to training on raw synthetic data?
• RQ3 To what extent can a model trained on one surgical system (MiroSurge) generalize to another (da Vinci) with different instrument shapes, textures, and colors?
• RQ4 How does the model perform under variations in instrument scale, position, and lighting conditions in real-world settings?

Key findings

• The proposed method achieved a mean Intersection over Union (IoU) of 88.06% on real endoscopic images of MiroSurge instruments, outperforming state-of-the-art methods in the literature.
• The model demonstrated strong zero-shot generalization to the da Vinci surgical system, achieving an average IoU of 50.28% despite significant domain shift in instrument shape, texture, and color.
• The model’s performance dropped significantly on da Vinci instruments, primarily due to differences in geometry and appearance, but still detected most of the instrument in several cases.
• The lowest IoU scores were consistently observed in images where the instrument occupied a disproportionately large portion of the image, indicating scale sensitivity.
• The method successfully reduced the need for manual annotation by leveraging synthetic data with perfect labels, transformed into realistic appearances via CycleGAN.
• The results suggest that fine-tuning the model on da Vinci data could significantly improve performance, though this was not pursued due to scope limitations.