[Paper Review] SurgMAE: Masked Autoencoders for Long Surgical Video Analysis

Muhammad Abdullah Jamal, Omid Mohareri|arXiv (Cornell University)|May 19, 2023

Surgical Simulation and TrainingMedicine3 citations

TL;DR

SurgMAE proposes a novel masked autoencoder framework for long surgical video analysis that improves self-supervised representation learning by sampling high spatio-temporal tokens for masking, rather than using random masking. It achieves state-of-the-art performance on OR-AR and OR-ARv2 datasets, especially under low-data regimes, and generalizes well to UCF-101, outperforming existing methods by up to 1.3% mAP and 0.9% top-1 accuracy.

ABSTRACT

There has been a growing interest in using deep learning models for processing long surgical videos, in order to automatically detect clinical/operational activities and extract metrics that can enable workflow efficiency tools and applications. However, training such models require vast amounts of labeled data which is costly and not scalable. Recently, self-supervised learning has been explored in computer vision community to reduce the burden of the annotation cost. Masked autoencoders (MAE) got the attention in self-supervised paradigm for Vision Transformers (ViTs) by predicting the randomly masked regions given the visible patches of an image or a video clip, and have shown superior performance on benchmark datasets. However, the application of MAE in surgical data remains unexplored. In this paper, we first investigate whether MAE can learn transferrable representations in surgical video domain. We propose SurgMAE, which is a novel architecture with a masking strategy based on sampling high spatio-temporal tokens for MAE. We provide an empirical study of SurgMAE on two large scale long surgical video datasets, and find that our method outperforms several baselines in low data regime. We conduct extensive ablation studies to show the efficacy of our approach and also demonstrate it's superior performance on UCF-101 to prove it's generalizability in non-surgical datasets as well.

Motivation & Objective

• To investigate whether masked autoencoders (MAE) can learn transferrable representations in long surgical video data.
• To address the limitations of random masking in MAE by proposing a novel token sampling strategy that prioritizes high information spatio-temporal regions.
• To improve data efficiency in surgical activity recognition by leveraging self-supervised pre-training on large-scale unlabeled surgical videos.
• To demonstrate the generalizability of the proposed method across surgical and non-surgical video benchmarks.

Proposed method

• Proposes SurgMAE, a Vision Transformer-based masked autoencoder adapted for long surgical video representation learning.
• Introduces a high spatio-temporal token sampling strategy that selects masked tokens based on embedding space distance, favoring informative regions and discarding redundant or background patches.
• Uses a high masking ratio (90% on OR-AR, 80% on UCF-101) to encourage robust feature learning.
• Employs a learnable mask token and a decoder head to reconstruct masked patches from visible tokens.
• Applies per-patch normalization and L2 (SE) loss for improved reconstruction quality.
• Pre-trains on large-scale unlabeled surgical video datasets (OR-ARv2, Cataract-101) before fine-tuning on downstream classification tasks.

Experimental results

Research questions

• RQ1 Can masked autoencoders effectively learn transferable representations from unlabeled long surgical videos?
• RQ2 Does sampling high spatio-temporal tokens for masking improve representation learning compared to random or frame/tube masking in surgical video MAE?
• RQ3 How does SurgMAE perform under low-data regimes compared to fully supervised and contrastive learning baselines?
• RQ4 Can the proposed method generalize to non-surgical video datasets like UCF-101?
• RQ5 What is the impact of different masking strategies on downstream surgical activity recognition performance?

Key findings

• SurgMAE achieves 68.91% mAP on OR-AR with only 5% labeled data, outperforming all baselines, including MAE with random masking (64.66%) and Swin-B+BiGRU (95.13% with full data).
• On the full OR-ARv2 dataset, SurgMAE achieves 93.11% mAP, surpassing the next best method (VideoMAE with tube masking at 92.36%).
• SurgMAE achieves 92.1% top-1 accuracy on UCF-101, outperforming VideoMAE (91.2%) under the same high masking ratio (80%).
• The proposed high spatio-temporal sampling strategy consistently improves performance across all datasets, confirming its efficacy in selecting informative tokens.
• Random masking still performs well under full-data fine-tuning, but SurgMAE’s advantage is most pronounced in low-data regimes, demonstrating superior data efficiency.
• Per-patch normalized pixel reconstruction with L2 (SE) loss yields better results than raw pixel reconstruction, aligning with findings in prior MAE work.