[Paper Review] Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition

Tong Yu, Didier Mutter+2 more|arXiv (Cornell University)|Nov 30, 2018

Surgical Simulation and Training19 references44 citations

TL;DR

The paper introduces a teacher/student semi-supervised framework for surgical phase recognition that uses a CNN-biLSTM-CRF teacher to generate synthetic labels for unannotated videos, training a real-time CNN-LSTM student, and shows improved performance with very scarce annotations.

ABSTRACT

Vision algorithms capable of interpreting scenes from a real-time video stream are necessary for computer-assisted surgery systems to achieve context-aware behavior. In laparoscopic procedures one particular algorithm needed for such systems is the identification of surgical phases, for which the current state of the art is a model based on a CNN-LSTM. A number of previous works using models of this kind have trained them in a fully supervised manner, requiring a fully annotated dataset. Instead, our work confronts the problem of learning surgical phase recognition in scenarios presenting scarce amounts of annotated data (under 25% of all available video recordings). We propose a teacher/student type of approach, where a strong predictor called the teacher, trained beforehand on a small dataset of ground truth-annotated videos, generates synthetic annotations for a larger dataset, which another model - the student - learns from. In our case, the teacher features a novel CNN-biLSTM-CRF architecture, designed for offline inference only. The student, on the other hand, is a CNN-LSTM capable of making real-time predictions. Results for various amounts of manually annotated videos demonstrate the superiority of the new CNN-biLSTM-CRF predictor as well as improved performance from the CNN-LSTM trained using synthetic labels generated for unannotated videos. For both offline and online surgical phase recognition with very few annotated recordings available, this new teacher/student strategy provides a valuable performance improvement by efficiently leveraging the unannotated data.

Motivation & Objective

• Address the challenge of surgical phase recognition with very limited manually annotated video data.
• Propose a teacher/student framework where a strong offline predictor generates synthetic labels for unannotated videos.
• Demonstrate that synthetic labels improve a real-time CNN-LSTM student’s performance and approach the fully supervised baseline.
• Compare offline and online inference capabilities within the same framework.

Proposed method

• Introduce a CNN-biLSTM-CRF teacher for offline inference to produce synthetic annotations.
• Use a ResNet-50 v2 CNN to extract 2048-d visual features from frames.
• Incorporate bidirectional LSTM to capture future context and a linear-chain CRF to model phase transitions.
• Train a CNN-LSTM student for real-time predictions using a mixed dataset of ground-truth and teacher-generated labels (G_{i,j}).
• Apply data augmentation and end-to-end training with backpropagation through time for the teacher.
• Evaluate multiple mini-training sets (sizes 1–80) on the cholec120 dataset with 7-phase labels.

Experimental results

Research questions

• RQ1 Can a teacher model trained on scarce annotations generate useful synthetic labels for unannotated videos in surgical phase recognition?
• RQ2 Does a teacher-generated labeling approach improve a real-time CNN-LSTM student compared to training only on scarce ground-truth data?
• RQ3 How close can semi-supervised training approach bring performance to fully supervised training as annotated data increases?
• RQ4 What is the impact of teacher architecture (CNN-biLSTM-CRF) versus simpler models on offline and online prediction performance?

Key findings

• The CNN-biLSTM-CRF teacher outperforms ablated variants and serves as the strongest predictor among offline models.
• CNN-LSTM trained with teacher-generated synthetic labels significantly improves over ground-truth-only training, reducing the data gap.
• With as few as 20 manually annotated videos, the CNN-biLSTM-CRF achieves 84.1% accuracy and 75.8% F1 on the test set, approaching the 89.5% accuracy and 82.5% F1 of fully supervised training.
• Teacher-generated annotations quality increases with more manually annotated videos, making G_{i,j} sets increasingly viable for training the student.
• Using synthetic labels, the gap between 20 and 80 ground-truth videos is halved for the CNN-LSTM online predictor, and can be fully closed if the student is replaced with an offline predictor.