[论文解读] The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methods

Vivek Singh Bawa, Gurkirt Singh+17 more|arXiv (Cornell University)|Apr 7, 2021

Surgical Simulation and Training参考文献 43被引用 29

一句话总结

本文介绍 ESAD，这是首个用于腹腔镜前列腺切除术中的医生动作检测的大规模数据集，分析基线模型和顶尖挑战方法，并讨论未来外科机器人研究的挑战与基准。

ABSTRACT

For an autonomous robotic system, monitoring surgeon actions and assisting the main surgeon during a procedure can be very challenging. The challenges come from the peculiar structure of the surgical scene, the greater similarity in appearance of actions performed via tools in a cavity compared to, say, human actions in unconstrained environments, as well as from the motion of the endoscopic camera. This paper presents ESAD, the first large-scale dataset designed to tackle the problem of surgeon action detection in endoscopic minimally invasive surgery. ESAD aims at contributing to increase the effectiveness and reliability of surgical assistant robots by realistically testing their awareness of the actions performed by a surgeon. The dataset provides bounding box annotation for 21 action classes on real endoscopic video frames captured during prostatectomy, and was used as the basis of a recent MIDL 2020 challenge. We also present an analysis of the dataset conducted using the baseline model which was released as part of the challenge, and a description of the top performing models submitted to the challenge together with the results they obtained. This study provides significant insight into what approaches can be effective and can be extended further. We believe that ESAD will serve in the future as a useful benchmark for all researchers active in surgeon action detection and assistive robotics at large.

研究动机与目标

• 介绍用于内镜下微创手术（MIS）中的外科医生动作检测的 ESAD 数据集。
• 为真实内镜帧定义带有边界框的注释协议和动作类别。
• 通过 SARAS-ESAD 挑战及基线模型建立基准评估框架。
• 识别外科动作检测的挑战与特征，以指导未来研究。

提出的方法

• 创建 ESAD，在真实根治性前列腺切除术的内镜帧上以边界框标注21个动作类别。
• 使用 VoTT 进行人工边界框标注，并定义严格指南以确保动作上下文标注（器官-工具靠近、30-70% 内容规则）。
• 发布一个基线单阶段检测器，基于特征金字塔网络（FPN），骨干为 ResNet，固定 BN 层。
• 尝试两种检测损失—Online Hard Example Mining（OHEM）和 focal loss，用以应对类别不平衡。
• 评估使用 IoU 阈值为 0.1、0.3 和 0.5 的平均精度均值（mAP）（Frame-mAP）。
• 提供实现细节和开源基线代码以实现复现实验。

实验结果

研究问题

• RQ1 在内镜视频中检测外科医生动作的方法学和实践挑战是什么？
• RQ2 ESAD 数据集如何在 MIS/R-MIS 情况下实现动作检测方法的基准评估？
• RQ3 在不同 IoU 阈值下，不同检测器架构和损失函数在 ESAD 上的表现如何？
• RQ4 类别不平衡和细粒度动作定义对检测性能有何影响？
• RQ5 基线方法和顶尖方法在验证集与测试集上的比较如何？

主要发现

• ESAD 包含四个 RARP 视频，在 21 个类别中共有 46,325 个动作实例。
• Training, validation, and test splits include 22,601 frames (28,055 instances), 4,574 frames (7,133 instances), and 6,223 frames (11,565 instances) respectively.
• 基线结果表明，在使用 OHEM 损失的情况下，增大图像尺寸可以提高验证准确率，但由于类别不平衡差异，这可能无法转化到测试集。
• 数据集显示高同类内变异性和低跨类变异性，使细粒度动作辨别具有挑战性。
• 注释指南要求边界框覆盖对象内容的 30-70%，并要求工具与器官上下文的接近程度以进行动作标注。