[Paper Review] Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networks

Gaurav Yengera, Didier Mutter+2 more|arXiv (Cornell University)|May 22, 2018

Reservoir Engineering and Simulation Methods37 references51 citations

TL;DR

Proposes a semi-supervised approach for surgical phase recognition by self-supervised pre-training CNN-LSTM networks on Remaining Surgery Duration (RSD); shows end-to-end training with RSD pre-training can reduce annotated data needs while maintaining or improving performance.

ABSTRACT

Real-time algorithms for automatically recognizing surgical phases are needed to develop systems that can provide assistance to surgeons, enable better management of operating room (OR) resources and consequently improve safety within the OR. State-of-the-art surgical phase recognition algorithms using laparoscopic videos are based on fully supervised training. This limits their potential for widespread application, since creation of manual annotations is an expensive process considering the numerous types of existing surgeries and the vast amount of laparoscopic videos available. In this work, we propose a new self-supervised pre-training approach based on the prediction of remaining surgery duration (RSD) from laparoscopic videos. The RSD prediction task is used to pre-train a convolutional neural network (CNN) and long short-term memory (LSTM) network in an end-to-end manner. Our proposed approach utilizes all available data and reduces the reliance on annotated data, thereby facilitating the scaling up of surgical phase recognition algorithms to different kinds of surgeries. Additionally, we present EndoN2N, an end-to-end trained CNN-LSTM model for surgical phase recognition and evaluate the performance of our approach on a dataset of 120 Cholecystectomy laparoscopic videos (Cholec120). This work also presents the first systematic study of self-supervised pre-training approaches to understand the amount of annotations required for surgical phase recognition. Interestingly, the proposed RSD pre-training approach leads to performance improvement even when all the training data is manually annotated and outperforms the single pre-training approach for surgical phase recognition presently published in the literature. It is also observed that end-to-end training of CNN-LSTM networks boosts surgical phase recognition performance.

Motivation & Objective

• Reduce reliance on manually annotated data for surgical phase recognition in laparoscopic videos.
• Leverage self-supervised pre-training to exploit unlabeled videos at scale.
• Promote end-to-end CNN-LSTM training to improve generalization across surgical phases and procedures.

Proposed method

• Introduce EndoN2N, an end-to-end CNN-LSTM model trained on complete video sequences with approximate backpropagation through time.
• Use self-supervised pre-training by predicting Remaining Surgery Duration (RSD) from laparoscopic videos to initialize CNN-LSTM models.
• Compare EndoN2N with EndoLSTM, a two-step CNN then LSTM training approach, on the same architecture.
• Provide an apples-to-apples comparison between end-to-end and two-step training for surgical phase recognition.
• Incorporate remaining surgery duration and progress as multi-task signals during RSD pre-training and during phase recognition.
• Partition long video sequences into subsequences for training due to memory constraints, with boundary state propagation.

Experimental results

Research questions

• RQ1 How does self-supervised RSD pre-training affect surgical phase recognition performance with varying amounts of annotated data?
• RQ2 Does end-to-end CNN-LSTM training outperform the two-step EndoLSTM approach for surgical phase recognition on complete video sequences?
• RQ3 Can RSD pre-training generalize across different surgeries and support scalable semi-supervised learning?

Key findings

• RSD pre-training enables similar or slightly better surgical phase recognition with about 20 percent fewer annotated videos.
• With about 50 percent fewer annotated videos, performance differences stay within about 5 percent.
• End-to-end training of CNN-LSTM networks (EndoN2N) improves performance and generalization compared to two-step training (EndoLSTM).
• RSD pre-training provides performance gains even when all training data are fully annotated, outperforming previously published single pre-training approaches for surgical phase recognition.
• Training CNN-LSTM end-to-end on long video sequences is feasible with the proposed approximation and yields improved results.