[论文解读] Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networks

Gaurav Yengera, Didier Mutter+2 more|arXiv (Cornell University)|May 22, 2018

Reservoir Engineering and Simulation Methods参考文献 37被引用 51

一句话总结

提出一种半监督方法用于外科手术阶段识别，通过对 Remaining Surgery Duration (RSD) 的自监督预训练 CNN-LSTM 网络；展示端到端训练结合 RSD 预训练可以在减少标注数据需求的同时保持或提升性能。

ABSTRACT

Real-time algorithms for automatically recognizing surgical phases are needed to develop systems that can provide assistance to surgeons, enable better management of operating room (OR) resources and consequently improve safety within the OR. State-of-the-art surgical phase recognition algorithms using laparoscopic videos are based on fully supervised training. This limits their potential for widespread application, since creation of manual annotations is an expensive process considering the numerous types of existing surgeries and the vast amount of laparoscopic videos available. In this work, we propose a new self-supervised pre-training approach based on the prediction of remaining surgery duration (RSD) from laparoscopic videos. The RSD prediction task is used to pre-train a convolutional neural network (CNN) and long short-term memory (LSTM) network in an end-to-end manner. Our proposed approach utilizes all available data and reduces the reliance on annotated data, thereby facilitating the scaling up of surgical phase recognition algorithms to different kinds of surgeries. Additionally, we present EndoN2N, an end-to-end trained CNN-LSTM model for surgical phase recognition and evaluate the performance of our approach on a dataset of 120 Cholecystectomy laparoscopic videos (Cholec120). This work also presents the first systematic study of self-supervised pre-training approaches to understand the amount of annotations required for surgical phase recognition. Interestingly, the proposed RSD pre-training approach leads to performance improvement even when all the training data is manually annotated and outperforms the single pre-training approach for surgical phase recognition presently published in the literature. It is also observed that end-to-end training of CNN-LSTM networks boosts surgical phase recognition performance.

研究动机与目标

• 减少腹腔镜视频中手术阶段识别对人工标注数据的依赖。
• 利用自监督预训练在大规模无标签视频中进行挖掘。
• 推动端到端的 CNN-LSTM 训练，以提升跨手术阶段和手术程序的泛化能力。

提出的方法

• 提出 EndoN2N，一种端到端的 CNN-LSTM 模型，在完整视频序列上进行训练，采用近似的时间反向传播（backpropagation through time）。
• 通过从腹腔镜视频中预测 Remaining Surgery Duration (RSD) 进行自监督预训练，用以初始化 CNN-LSTM 模型。
• 在相同架构下，将 EndoN2N 与 EndoLSTM（先训练 CNN 再训练 LSTM 的两步方法）进行比较。
• 提供端到端与两步训练在手术阶段识别任务上的可比对对比。
• 在 RSD 预训练和阶段识别阶段，将 remaining surgery duration 和进度作为多任务信号纳入。
• 由于内存受限，将长视频序列划分为子序列以进行训练，并进行边界状态传播。

实验结果

研究问题

• RQ1 自监督 RSD 预训练在标注数据数量变化下对手术阶段识别性能的影响如何？
• RQ2 在完整视频序列上，端到端的 CNN-LSTM 训练是否优于两步的 EndoLSTM 方法？
• RQ3 RSD 预训练能否在不同手术之间实现泛化，并支持可扩展的半监督学习？

主要发现

• RSD 预训练在大约减少 20% 标注视频的情况下，能够实现相似或略微更好的手术阶段识别。
• 在大约减少 50% 标注视频的情况下，性能差异保持在约 5% 内。
• 端到端训练 CNN-LSTM（EndoN2N）在性能和泛化方面优于两步训练（EndoLSTM）。
• 即使所有训练数据均已完全标注，RSD 预训练仍带来性能提升，超过先前公开的单一预训练方法在手术阶段识别上的表现。
• 在长视频序列上进行端到端的 CNN-LSTM 训练在所提出的近似下是可行的，并产生改进的结果。