[论文解读] Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures

Sanat Ramesh, Diego Dall’Alba+7 more|arXiv (Cornell University)|Feb 24, 2021

Surgical Simulation and Training参考文献 31被引用 84

一句话总结

该论文提出了一种多任务多阶段时间卷积网络（MTMS-TCN），用于在腹腔镜胃旁路手术的内窥镜视频中联合识别手术阶段和步骤。通过利用共享时间表征的多任务学习，该模型实现了最先进性能，在新发布的Bypass40数据集上，阶段识别的准确率、精确率和召回率相比单任务模型和基于LSTM的模型提高了1–2%，步骤识别的准确率提高了3–6%。

ABSTRACT

Purpose: Automatic segmentation and classification of surgical activity is crucial for providing advanced support in computer-assisted interventions and autonomous functionalities in robot-assisted surgeries. Prior works have focused on recognizing either coarse activities, such as phases, or fine-grained activities, such as gestures. This work aims at jointly recognizing two complementary levels of granularity directly from videos, namely phases and steps. Method: We introduce two correlated surgical activities, phases and steps, for the laparoscopic gastric bypass procedure. We propose a Multi-task Multi-Stage Temporal Convolutional Network (MTMS-TCN) along with a multi-task Convolutional Neural Network (CNN) training setup to jointly predict the phases and steps and benefit from their complementarity to better evaluate the execution of the procedure. We evaluate the proposed method on a large video dataset consisting of 40 surgical procedures (Bypass40). Results: We present experimental results from several baseline models for both phase and step recognition on the Bypass40 dataset. The proposed MTMS-TCN method outperforms in both phase and step recognition by 1-2% in accuracy, precision and recall, compared to single-task methods. Furthermore, for step recognition, MTMS-TCN achieves a superior performance of 3-6% compared to LSTM based models in accuracy, precision, and recall. Conclusion: In this work, we present a multi-task multi-stage temporal convolutional network for surgical activity recognition, which shows improved results compared to single-task models on the Bypass40 gastric bypass dataset with multi-level annotations. The proposed method shows that the joint modeling of phases and steps is beneficial to improve the overall recognition of each type of activity.

研究动机与目标

• 解决腹腔镜胃旁路手术中缺乏多层级手术活动标注的问题。
• 开发一种用于手术阶段和步骤联合识别的框架，以提升整体手术流程理解能力。
• 评估多任务学习与时间卷积网络在手术视频理解中的有效性。
• 基于包含40个标注视频、11个阶段和44个步骤的Bypass40数据集，建立新的基准。

提出的方法

• 提出一种多任务多阶段时间卷积网络（MTMS-TCN），从内窥镜视频中联合预测手术阶段和步骤。
• 采用多阶段架构，每一阶段通过空洞时间卷积对前一阶段的预测结果进行细化。
• 使用共享的CNN主干网络（ResNet）进行视觉特征提取，随后通过针对任务的分类头完成阶段和步骤分类。
• 应用因果卷积，实现适用于实时手术监测的在线序列预测。
• 采用多任务训练设置，共享特征并使用任务特定损失，以挖掘阶段与步骤之间的互补关系。
• 采用4折交叉验证，并在Bypass40数据集上使用准确率、精确率、召回率和F1分数评估性能。

实验结果

研究问题

• RQ1 与单任务学习相比，联合建模手术阶段和步骤是否能提升识别性能？
• RQ2 与基于LSTM的模型相比，多任务时间卷积网络在识别细粒度手术步骤和粗粒度阶段方面表现如何？
• RQ3 多任务学习在识别持续时间短或手术关键的步骤方面能提升多少？
• RQ4 阶段与步骤之间的层次关系是否有助于提升模型的整体泛化能力和时间推理能力？

主要发现

• MTMS-TCN在阶段识别中达到91.16%的准确率，比表现第二好的模型（TeCNO）高出2%。
• 在步骤识别中，MTMS-TCN达到76.09%的准确率，比TeCNO高出1%，比基于LSTM的模型在精确率和召回率上高出3–6%。
• 该模型显著提升了对短时步骤（如S25、S30和S39）的识别能力，这些步骤对其他模型而言极具挑战。
• MTMS-TCN的联合识别准确率（75.14%）非常接近其步骤识别准确率，表明其有效建模了层次关系。
• 多任务设置显著提升了两个任务的性能，MT-ResNet（主干网络）在步骤识别上优于ResNet，尽管阶段识别准确率略有下降。
• 可视化结果表明，MTMS-TCN在捕捉阶段转换以及小阶段（如P5、P7、P9）方面优于TeCNO和ResNetLSTM。