[论文解读] SAM Meets Robotic Surgery: An Empirical Study on Generalization, Robustness and Adaptation

An Wang, Mobarakol Islam+3 more|arXiv (Cornell University)|Aug 14, 2023

Artificial Intelligence in Healthcare and EducationMedicine被引用 3

一句话总结

本研究评估了分割一切模型（SAM）在机器人手术器械分割中的表现，揭示了其在复杂条件下（如器械重叠、反光和数据损坏）零样本泛化能力的局限性。为解决这些问题，作者通过LoRA微调提出了SurgicalSAM，实现了无需提示的最先进性能，展示了类别感知的掩码预测能力，并在手术场景中提升了鲁棒性。

ABSTRACT

The Segment Anything Model (SAM) serves as a fundamental model for semantic segmentation and demonstrates remarkable generalization capabilities across a wide range of downstream scenarios. In this empirical study, we examine SAM's robustness and zero-shot generalizability in the field of robotic surgery. We comprehensively explore different scenarios, including prompted and unprompted situations, bounding box and points-based prompt approaches, as well as the ability to generalize under corruptions and perturbations at five severity levels. Additionally, we compare the performance of SAM with state-of-the-art supervised models. We conduct all the experiments with two well-known robotic instrument segmentation datasets from MICCAI EndoVis 2017 and 2018 challenges. Our extensive evaluation results reveal that although SAM shows remarkable zero-shot generalization ability with bounding box prompts, it struggles to segment the whole instrument with point-based prompts and unprompted settings. Furthermore, our qualitative figures demonstrate that the model either failed to predict certain parts of the instrument mask (e.g., jaws, wrist) or predicted parts of the instrument as wrong classes in the scenario of overlapping instruments within the same bounding box or with the point-based prompt. In fact, SAM struggles to identify instruments in complex surgical scenarios characterized by the presence of blood, reflection, blur, and shade. Additionally, SAM is insufficiently robust to maintain high performance when subjected to various forms of data corruption. We also attempt to fine-tune SAM using Low-rank Adaptation (LoRA) and propose SurgicalSAM, which shows the capability in class-wise mask prediction without prompt. Therefore, we can argue that, without further domain-specific fine-tuning, SAM is not ready for downstream surgical tasks.

研究动机与目标

• 评估SAM在真实世界内窥镜数据集上进行机器人手术时的零样本泛化能力与鲁棒性。
• 研究SAM在各种提示（边界框、点提示）及无提示设置下的表现。
• 评估SAM在模糊、血液、反光等数据损坏和手术场景复杂性下的抗干扰能力。
• 开发并验证一个领域自适应的SAM版本——SurgicalSAM，实现无需提示的类别感知分割。
• 将SurgicalSAM与最先进监督模型在器械分割任务中进行基准对比。

提出的方法

• 通过低秩适应（LoRA）对基于ViT-H的SAM进行微调，使其在无需提示输入的情况下适应手术器械分割任务。
• 以MICCAI EndoVis数据集（2017年和2018年）作为微调基础，使用人工标注的边界框和中心点提示进行评估。
• 采用AdamW优化器、权重衰减、512×512输入分辨率及标准交叉熵损失函数，对SurgicalSAM进行二值分割与器械类别分割训练。
• 通过EndoVis18验证集在五种严重程度的合成损坏（如模糊、噪声、雾霾、JPEG伪影）下评估鲁棒性。
• 使用IoU和Dice指标与14种最先进模型（包括UNet、ISINet和S3Net）进行定量比较。
• 通过消融实验分析提示类型与损坏级别，以识别原始SAM在复杂手术场景中的失败模式。

实验结果

研究问题

• RQ1 SAM在未微调的情况下能否有效泛化至机器人手术器械分割任务，特别是在点提示或无提示设置下？
• RQ2 SAM在真实手术图像损坏（如模糊、噪声、雾霾和运动模糊）下的表现如何？
• RQ3 当器械重叠或出现复杂纹理（如反光与阴影）时，SAM的失败模式是什么？
• RQ4 基于LoRA的微调能否使SAM在手术场景中实现类别感知、无提示的分割？
• RQ5 微调后的SurgicalSAM在mIoU和鲁棒性方面与最先进监督模型相比表现如何？

主要发现

• 在EndoVis18数据集上，SAM使用边界框提示时mIoU达到71.38%，表明在简单场景中具备强大的零样本泛化能力。
• 在点提示或无提示设置下，SAM无法完整分割器械，常遗漏钳口或腕部等部分。
• 在数据损坏下性能显著下降，高斯噪声最严重级别下mIoU降至3.79%，模糊损坏下降至4.26%。
• SurgicalSAM在EndoVis18验证集上无需任何提示即达到71.38%的mIoU，与最先进模型性能相当，尽管其为单任务、仅图像输入架构。
• SurgicalSAM仅需额外18.28MB参数，表明LoRA微调策略具有极高效率。
• 模型在模糊和运动损坏下表现出明显的鲁棒性下降，表明仍需进一步领域自适应。