[Paper Review] MediViSTA: Medical Video Segmentation via Temporal Fusion SAM Adaptation for Echocardiography

Sekeun Kim, Jin, Pengfei+10 more|arXiv (Cornell University)|Sep 24, 2023

Advanced Neural Network ApplicationsComputer Science3 citations

TL;DR

This paper proposes MediViSTA-SAM, a novel spatio-temporal adapter framework that adapts the Segmentation Anything Model (SAM) for zero-shot medical video segmentation in echocardiography. By integrating cross-frame attention, multi-scale U-shaped encoding, and a modified mask decoder, it achieves state-of-the-art performance with a 87.0% Dice score and 0.06 temporal smoothness on multi-vendor echocardiography datasets.

ABSTRACT

Despite achieving impressive results in general-purpose semantic segmentation with strong generalization on natural images, the Segment Anything Model (SAM) has shown less precision and stability in medical image segmentation. In particular, the original SAM architecture is designed for 2D natural images and is therefore not support to handle three-dimensional information, which is particularly important for medical imaging modalities that are often volumetric or video data. In this paper, we introduce MediViSTA, a parameter-efficient fine-tuning method designed to adapt the vision foundation model for medical video, with a specific focus on echocardiographic segmentation. To achieve spatial adaptation, we propose a frequency feature fusion technique that injects spatial frequency information from a CNN branch. For temporal adaptation, we integrate temporal adapters within the transformer blocks of the image encoder. Using a fine-tuning strategy, only a small subset of pre-trained parameters is updated, allowing efficient adaptation to echocardiographic data. The effectiveness of our method has been comprehensively evaluated on three datasets, comprising two public datasets and one multi-center in-house dataset. Our method consistently outperforms various state-of-the-art approaches without using any prompts. Furthermore, our model exhibits strong generalization capabilities on unseen datasets, surpassing the second-best approach by 2.15\% in Dice and 0.09 in temporal consistency. The results demonstrate the potential of MediViSTA to significantly advance echocardiographical video segmentation, offering improved accuracy and robustness in cardiac assessment applications.

Motivation & Objective

• Address the domain gap between natural image datasets (used to pre-train SAM) and medical imaging modalities, particularly echocardiography.
• Overcome SAM's poor performance in medical video segmentation due to differences in image physics, artifacts, and multi-vendor data distribution.
• Enable zero-shot generalization in echocardiography video segmentation without fine-tuning on target medical data.
• Integrate both spatial multi-scale feature learning and long/short-range temporal modeling to improve boundary delineation and temporal consistency.
• Develop a parameter-efficient adaptation framework that preserves pre-trained SAM weights while enabling effective medical video understanding.

Proposed method

• Propose a spatio-temporal adapter that applies cross-frame attention, using the immediately preceding frame as a reference to model temporal dynamics.
• Implement a U-shaped encoder with multi-scale feature extraction to capture objects of varying sizes in echocardiographic images.
• Introduce a modified mask decoder to enhance segmentation accuracy across diverse anatomical structures.
• Use a two-branch attention mechanism where temporal and spatial attention are applied sequentially (temporal first, then spatial) for optimal feature fusion.
• Leverage pre-trained ViT-based SAM (vit_h, vit_l, vit_b) as the backbone, preserving weights to enable zero-shot transfer learning.
• Apply parameter-efficient adaptation by freezing the original SAM weights and only training the adapter and decoder components.

Figure 1: Image-to-video transfer learning strategies. (a)Vanilla transformer block (b) 3D adapter with depthwise 3D convolution [ 20 ] (c) 2D image to 3D adapter [ 33 ] (d) Proposed medical video spatio-temporal adapter (MediViSTA).

Experimental results

Research questions

• RQ1 Can the Segmentation Anything Model (SAM) be effectively adapted for zero-shot medical video segmentation in echocardiography?
• RQ2 How does incorporating cross-frame attention with a reference frame improve temporal consistency and segmentation accuracy?
• RQ3 What is the contribution of multi-scale feature fusion and a modified mask decoder to performance on small and variable-sized cardiac structures?
• RQ4 How does the order of temporal and spatial attention operations affect segmentation outcomes in medical video?
• RQ5 To what extent does using a pre-trained SAM backbone improve zero-shot generalization compared to training from scratch?

Key findings

• The proposed MediViSTA-SAM framework achieves a Dice score of 87.0% and temporal smoothness of 0.06, outperforming all ablated variants and state-of-the-art methods.
• The ablation study confirms that multi-scale fusion contributes significantly, with a 1.1% drop in Dice when removed, and increases temporal smoothness by 0.03.
• Cross-frame attention improves performance by 0.2% in Dice compared to the baseline without it, with a notable reduction in temporal smoothness from 0.09 to 0.06.
• Using the largest pre-trained SAM backbone (vit_h) yields the highest Dice (87.0%) and lowest temporal smoothness (0.06), confirming the benefit of larger models.
• Training without pre-trained SAM weights results in a 4.42% drop in Dice score, demonstrating the critical role of pre-training for downstream medical segmentation.
• The full model with both spatio-temporal adapter and multi-scale fusion achieves the best performance (87.0% Dice, 0.06 L), confirming synergistic benefits of all components.

Figure 2: The overview of MedViSTA-SAM, which consists of long and short range cross-frame attention and spatial transformer with U-shaped framework for medical video segmentation. We need reshape input [B, T, C, H, W] to [BT, C, H, W ] for our framework. During training, blue part are frozen while