[Paper Review] TUNeS: A Temporal U-Net with Self-Attention for Video-based Surgical Phase Recognition

Isabel Funke, Dominik Rivoir+2 more|arXiv (Cornell University)|Jul 19, 2023

Surgical Simulation and Training4 citations

TL;DR

TUNeS proposes a novel Temporal U-Net with self-attention for video-based surgical phase recognition, integrating self-attention within a U-Net architecture to model long-range temporal dependencies efficiently. By jointly training a CNN feature extractor with an LSTM on long video segments (up to 64 frames), TUNeS achieves state-of-the-art performance on the Cholec80 dataset, attaining 92.3% accuracy and 78.4% Macro Jaccard in offline mode with L=64 context.

ABSTRACT

Objective: To enable context-aware computer assistance in the operating room of the future, cognitive systems need to understand automatically which surgical phase is being performed by the medical team. The primary source of information for surgical phase recognition is typically video, which presents two challenges: extracting meaningful features from the video stream and effectively modeling temporal information in the sequence of visual features. Methods: For temporal modeling, attention mechanisms have gained popularity due to their ability to capture long-range dependencies. In this paper, we explore design choices for attention in existing temporal models for surgical phase recognition and propose a novel approach that uses attention more effectively and does not require hand-crafted constraints: TUNeS, an efficient and simple temporal model that incorporates self-attention at the core of a convolutional U-Net structure. In addition, we propose to train the feature extractor, a standard CNN, together with an LSTM on preferably long video segments, i.e., with long temporal context. Results: In our experiments, almost all temporal models performed better on top of feature extractors that were trained with longer temporal context. On these contextualized features, TUNeS achieves state-of-the-art results on the Cholec80 dataset. Conclusion: This study offers new insights on how to use attention mechanisms to build accurate and efficient temporal models for surgical phase recognition. Significance: Implementing automatic surgical phase recognition is essential to automate the analysis and optimization of surgical workflows and to enable context-aware computer assistance during surgery, thus ultimately improving patient care.

Motivation & Objective

• To improve temporal modeling in video-based surgical phase recognition by effectively integrating self-attention into a convolutional U-Net architecture.
• To investigate the impact of training the visual feature extractor with long temporal context (up to 64 frames) on subsequent phase recognition performance.
• To develop a model that enables both online and offline surgical phase recognition with high accuracy and robustness to ambiguous frames.
• To demonstrate that self-attention can be applied efficiently in a U-Net structure without hand-crafted constraints, enhancing long-range dependency modeling.
• To establish a new state-of-the-art benchmark on the Cholec80 dataset using contextualized features and a unified attention-enhanced temporal model.

Proposed method

• Proposes TUNeS, a Temporal U-Net that embeds self-attention at the bottleneck of a U-Net encoder-decoder structure to model long-range temporal dependencies in downsampled, semantically rich features.
• Integrates a CNN feature extractor with an LSTM in a joint training setup on truncated video sequences of length L (1, 8, or 64 frames), enabling the CNN to learn context-aware representations.
• Applies self-attention after temporal downsampling to reduce computational cost while preserving global context modeling capability.
• Uses a U-Net-like skip-connection architecture to preserve spatial and temporal resolution, enhancing feature reconstruction and phase prediction accuracy.
• Employs a cosine annealing learning rate scheduler to improve generalization and convergence during training.
• Trains the model end-to-end on the Cholec80 dataset using standard cross-entropy and smoothing losses, with inference performed using only the trained CNN and TUNeS backbone.

Experimental results

Research questions

• RQ1 Does training the visual feature extractor with long temporal context (L=64) significantly improve surgical phase recognition performance compared to single-frame or short-sequence training?
• RQ2 Can self-attention be effectively and efficiently integrated into a U-Net-based architecture for temporal modeling in surgical video analysis?
• RQ3 How does TUNeS compare to existing state-of-the-art models in terms of accuracy and robustness across different evaluation modes (online and offline)?
• RQ4 To what extent does the use of self-attention in the U-Net bottleneck improve long-range temporal dependency modeling in surgical workflows?
• RQ5 Does the joint training of CNN and LSTM on long video segments lead to more discriminative and context-aware features for phase recognition?

Key findings

• TUNeS achieves a test accuracy of 92.3% and a Macro Jaccard score of 78.4% on the Cholec80 dataset in offline mode when trained with a temporal context length of L=64.
• The model with L=64 context achieves the highest performance across all settings, demonstrating that long-context training significantly improves feature quality and recognition accuracy.
• TUNeS outperforms all baseline models, including TeCNO, ASFormer, and SAHC, on both online and offline evaluation protocols, establishing a new state-of-the-art on Cholec80.
• The confusion matrices show high diagonal values in both row-normalized (recall) and column-normalized (precision) forms, indicating strong per-phase classification performance and low misclassification rates.
• The use of a cosine annealing learning rate scheduler further improves performance, with TUNeS achieving 94.6% accuracy and 82.5% Macro Jaccard in offline mode when combined with this scheduling strategy.
• All temporal models, including TUNeS, perform better when features are extracted from a CNN jointly trained with an LSTM on long video segments, confirming the benefit of contextualized feature learning.