[Paper Review] Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image Analysis

Moein Heidari, Sina Ghorbani Kolahi+9 more|arXiv (Cornell University)|Jun 5, 2024

Neural Networks and Applications4 citations

TL;DR

This survey provides a comprehensive review of State Space Models (SSMs), particularly the Mamba architecture, in medical image analysis. It systematically categorizes Mamba-based methods across imaging modalities, organs, and tasks such as segmentation, classification, and multi-modal fusion, highlighting their computational efficiency and potential to overcome limitations of CNNs and Transformers in long-range dependency modeling and resource-constrained settings.

ABSTRACT

Sequence modeling plays a vital role across various domains, with recurrent neural networks being historically the predominant method of performing these tasks. However, the emergence of transformers has altered this paradigm due to their superior performance. Built upon these advances, transformers have conjoined CNNs as two leading foundational models for learning visual representations. However, transformers are hindered by the $\mathcal{O}(N^2)$ complexity of their attention mechanisms, while CNNs lack global receptive fields and dynamic weight allocation. State Space Models (SSMs), specifically the extit{ extbf{Mamba}} model with selection mechanisms and hardware-aware architecture, have garnered immense interest lately in sequential modeling and visual representation learning, challenging the dominance of transformers by providing infinite context lengths and offering substantial efficiency maintaining linear complexity in the input sequence. Capitalizing on the advances in computer vision, medical imaging has heralded a new epoch with Mamba models. Intending to help researchers navigate the surge, this survey seeks to offer an encyclopedic review of Mamba models in medical imaging. Specifically, we start with a comprehensive theoretical review forming the basis of SSMs, including Mamba architecture and its alternatives for sequence modeling paradigms in this context. Next, we offer a structured classification of Mamba models in the medical field and introduce a diverse categorization scheme based on their application, imaging modalities, and targeted organs. Finally, we summarize key challenges, discuss different future research directions of the SSMs in the medical domain, and propose several directions to fulfill the demands of this field. In addition, we have compiled the studies discussed in this paper along with their open-source implementations on our GitHub repository.

Motivation & Objective

• To provide a systematic, up-to-date review of State Space Models (SSMs), especially Mamba, in medical image analysis.
• To address the limitations of CNNs in capturing long-range dependencies and Transformers in computational complexity.
• To classify and analyze Mamba-based methods according to application domain, imaging modality, and target anatomy.
• To identify key challenges such as explainability, performance gaps in classification, and lack of foundation models in medical imaging.
• To propose future research directions, including multi-modal learning, medical foundation models, and improved scanning schemes for 2D/3D data.

Proposed method

• The survey conducts a theoretical review of SSMs, focusing on Mamba’s selective state space mechanism and hardware-aware architecture.
• It introduces a novel taxonomy of Mamba models based on application type (e.g., segmentation, classification), imaging modality (MRI, CT, X-ray), and target organ.
• The authors analyze Mamba’s linear complexity (O(N)) in sequence length, contrasting it with the O(N²) attention complexity of Transformers.
• The study evaluates Mamba’s performance across medical imaging tasks, including image reconstruction, registration, and multi-modal understanding.
• It examines architectural innovations such as Mamba-2, which integrates SSMs with attention mechanisms to improve performance.
• The survey compiles open-source implementations and relevant papers in a regularly updated GitHub repository for reproducibility and accessibility.

Figure 1: Algorithms and methods trilemma. Despite CNN’s ability to efficiently process local information with high computational efficiency, they often struggle with capturing long-range dependencies due to their inherent architectural constraints. On the other hand, transformers excel at capturing

Experimental results

Research questions

• RQ1 How do Mamba models compare to CNNs and Transformers in modeling long-range dependencies in medical images while maintaining computational efficiency?
• RQ2 What are the key architectural components of Mamba that enable its performance in medical image analysis tasks?
• RQ3 In which medical imaging applications (e.g., segmentation, classification, reconstruction) have Mamba models shown the most promise?
• RQ4 What are the major challenges hindering Mamba’s adoption in clinical AI, including explainability and performance gaps in high-level vision tasks?
• RQ5 What future research directions are most promising for advancing Mamba-based models in medical imaging, particularly in multi-modal learning and foundation models?

Key findings

• Mamba models achieve linear complexity (O(N)) in sequence length, offering a significant efficiency advantage over Transformers’ O(N²) self-attention mechanism.
• Despite strong performance in autoregressive and long-sequence tasks, Mamba models currently underperform SOTA CNN and ViT models in image classification tasks.
• The Mamba-2 architecture, which combines SSMs with attention, shows improved performance by theoretically bridging the gap between SSMs and attention mechanisms.
• Mamba models show strong potential in tasks like image segmentation, reconstruction, and registration, where long-range spatial dependencies are critical.
• There remains a significant research gap in developing Mamba-based foundation models and multi-modal learning frameworks tailored for medical imaging.
• Explainability of Mamba in visual tasks remains limited, with only partial theoretical and empirical insights into its decision-making processes in medical image analysis.

(a) Applications