[Paper Review] Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures

Sanat Ramesh, Diego Dall’Alba+7 more|arXiv (Cornell University)|Feb 24, 2021

Surgical Simulation and Training31 references84 citations

TL;DR

This paper proposes a Multi-Task Multi-Stage Temporal Convolutional Network (MTMS-TCN) for joint recognition of surgical phases and steps in laparoscopic gastric bypass procedures using endoscopic video. By leveraging multi-task learning with shared temporal representations, the model achieves state-of-the-art performance, outperforming single-task and LSTM-based models by 1–2% in accuracy, precision, and recall for phase recognition, and by 3–6% for step recognition on the new Bypass40 dataset with hierarchical annotations.

ABSTRACT

Purpose: Automatic segmentation and classification of surgical activity is crucial for providing advanced support in computer-assisted interventions and autonomous functionalities in robot-assisted surgeries. Prior works have focused on recognizing either coarse activities, such as phases, or fine-grained activities, such as gestures. This work aims at jointly recognizing two complementary levels of granularity directly from videos, namely phases and steps. Method: We introduce two correlated surgical activities, phases and steps, for the laparoscopic gastric bypass procedure. We propose a Multi-task Multi-Stage Temporal Convolutional Network (MTMS-TCN) along with a multi-task Convolutional Neural Network (CNN) training setup to jointly predict the phases and steps and benefit from their complementarity to better evaluate the execution of the procedure. We evaluate the proposed method on a large video dataset consisting of 40 surgical procedures (Bypass40). Results: We present experimental results from several baseline models for both phase and step recognition on the Bypass40 dataset. The proposed MTMS-TCN method outperforms in both phase and step recognition by 1-2% in accuracy, precision and recall, compared to single-task methods. Furthermore, for step recognition, MTMS-TCN achieves a superior performance of 3-6% compared to LSTM based models in accuracy, precision, and recall. Conclusion: In this work, we present a multi-task multi-stage temporal convolutional network for surgical activity recognition, which shows improved results compared to single-task models on the Bypass40 gastric bypass dataset with multi-level annotations. The proposed method shows that the joint modeling of phases and steps is beneficial to improve the overall recognition of each type of activity.

Motivation & Objective

• To address the lack of multi-level surgical activity annotations in laparoscopic gastric bypass procedures.
• To develop a joint recognition framework for surgical phases and steps to improve overall workflow understanding.
• To evaluate the effectiveness of multi-task learning with temporal convolutional networks on surgical video understanding.
• To establish a new benchmark using the Bypass40 dataset with 40 annotated videos and 11 phases/44 steps.

Proposed method

• Proposes a Multi-Task Multi-Stage Temporal Convolutional Network (MTMS-TCN) that jointly predicts surgical phases and steps from endoscopic video.
• Employs a multi-stage architecture where each stage refines predictions from the previous one using dilated temporal convolutions.
• Uses a shared CNN backbone (ResNet) for visual feature extraction followed by task-specific heads for phase and step classification.
• Applies causal convolutions to enable online, sequential prediction suitable for real-time surgical monitoring.
• Implements a multi-task training setup with shared features and task-specific losses to exploit complementary relationships between phases and steps.
• Utilizes 4-fold cross-validation and evaluates performance using accuracy, precision, recall, and F1-score on the Bypass40 dataset.

Experimental results

Research questions

• RQ1 Can joint modeling of surgical phases and steps improve recognition performance compared to single-task learning?
• RQ2 How does a multi-task temporal convolutional network compare to LSTM-based models in recognizing fine-grained surgical steps and coarse phases?
• RQ3 To what extent does multi-task learning enhance recognition of short-duration or surgically critical steps?
• RQ4 Does the hierarchical relationship between phases and steps benefit the model's overall generalization and temporal reasoning?

Key findings

• MTMS-TCN achieves 91.16% accuracy in phase recognition, outperforming the next best model (TeCNO) by 2%.
• For step recognition, MTMS-TCN reaches 76.09% accuracy, surpassing TeCNO by 1% and LSTM-based models by 3–6% in precision and recall.
• The model improves recognition of short-duration steps such as S25, S30, and S39, which are challenging for other models.
• MTMS-TCN's joint recognition accuracy (75.14%) is very close to its step recognition accuracy, indicating effective modeling of hierarchical relationships.
• The multi-task setup leads to improved performance on both tasks, with MT-ResNet (backbone) showing better step recognition than ResNet despite a slight drop in phase accuracy.
• Visualizations confirm that MTMS-TCN better captures phase transitions and smaller phases (e.g., P5, P7, P9) than TeCNO and ResNetLSTM.