[Paper Review] MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation

Nicolás Ayobi, Alejandra Pérez-Rondón+2 more|arXiv (Cornell University)|Mar 16, 2023

Surgical Simulation and TrainingMedicine34 references3 citations

TL;DR

MATIS proposes a fully transformer-based method for surgical instrument segmentation using masked and deformable attention to generate precise instance-level region proposals, enhanced by a video transformer module for temporal consistency. It achieves state-of-the-art performance on Endovis 2017 and 2018, with mIoU of 84.26 and 82.37 respectively, outperforming prior methods through improved classification and temporal coherence.

ABSTRACT

We propose Masked-Attention Transformers for Surgical Instrument Segmentation (MATIS), a two-stage, fully transformer-based method that leverages modern pixel-wise attention mechanisms for instrument segmentation. MATIS exploits the instance-level nature of the task by employing a masked attention module that generates and classifies a set of fine instrument region proposals. Our method incorporates long-term video-level information through video transformers to improve temporal consistency and enhance mask classification. We validate our approach in the two standard public benchmarks, Endovis 2017 and Endovis 2018. Our experiments demonstrate that MATIS' per-frame baseline outperforms previous state-of-the-art methods and that including our temporal consistency module boosts our model's performance further.

Motivation & Objective

• To address the limitations of pixel-wise classification in surgical instrument segmentation, particularly spatial inconsistency and poor handling of instance-level distinctions.
• To improve segmentation accuracy by leveraging masked and deformable attention mechanisms for fine-grained region proposal generation.
• To enhance temporal consistency in video sequences by integrating long-term video-level features through a TAPIR-inspired architecture.
• To establish a new state-of-the-art in surgical instrument segmentation by combining instance-aware mask classification with video-level reasoning.

Proposed method

• MATIS employs a Mask2Former-based backbone with a Swin Transformer encoder to generate multi-scale, deformable attention features for precise mask prediction.
• It uses a class-specific inference strategy that selects top-k masks per instrument category using confidence thresholds, improving instance-level accuracy.
• A temporal consistency module integrates video-level features via a TAPIR-like architecture, pooling features across a sequence of frames to refine mask classification.
• The model incorporates a time-specific MLP to project pooled temporal features for better alignment with segment embeddings, enhancing joint classification.
• It applies multi-label classification supervision on time features to guide the model in recognizing instrument presence across frames.
• The method combines per-frame mask classification with long-range video modeling, enabling robust and consistent segmentation across surgical video sequences.

Experimental results

Research questions

• RQ1 Can a fully transformer-based architecture with masked and deformable attention outperform CNN-based models in surgical instrument instance segmentation?
• RQ2 How does incorporating long-term video-level context improve temporal consistency and segmentation accuracy in endoscopic surgical videos?
• RQ3 What is the optimal inference strategy for selecting instance masks when multiple proposals are generated per frame?
• RQ4 To what extent does joint modeling of temporal and spatial features enhance classification performance compared to frame-by-frame inference?
• RQ5 How do architectural components like time-specific MLPs and presence supervision contribute to improved mask classification over time?

Key findings

• MATIS achieves a mIoU of 84.26 on Endovis 2018, representing a new state-of-the-art performance for surgical instrument segmentation.
• The inclusion of the temporal consistency module, with both Time MLP and presence supervision, improves mIoU by 1.05 points compared to the baseline.
• The best inference strategy—top-k per-class masks with per-class confidence thresholds—achieves 82.37 mIoU, outperforming general filtering or NMS.
• Performance degrades with smaller input window sizes, indicating the importance of long-term video context for robust segmentation.
• The use of class-specific thresholds and per-class mask selection significantly improves performance on rare instrument categories with fewer training instances.
• The ablation study confirms that both the time-specific MLP and multi-label presence supervision are essential for maximizing temporal consistency and classification accuracy.