[Paper Review] Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review

Reza Azad, Amirhossein Kazerouni+7 more|arXiv (Cornell University)|Jan 9, 2023

COVID-19 diagnosis using AI11 citations

TL;DR

A systematic encyclopedia of Transformer-based methods for medical image analysis, covering classification, segmentation, detection, registration, reconstruction, synthesis, and report generation, with taxonomies, benchmarks, and future directions.

ABSTRACT

The remarkable performance of the Transformer architecture in natural language processing has recently also triggered broad interest in Computer Vision. Among other merits, Transformers are witnessed as capable of learning long-range dependencies and spatial correlations, which is a clear advantage over convolutional neural networks (CNNs), which have been the de facto standard in Computer Vision problems so far. Thus, Transformers have become an integral part of modern medical image analysis. In this review, we provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks, including classification, segmentation, detection, registration, synthesis, and clinical report generation. For each of these applications, we investigate the novelty, strengths and weaknesses of the different proposed strategies and develop taxonomies highlighting key properties and contributions. Further, if applicable, we outline current benchmarks on different datasets. Finally, we summarize key challenges and discuss different future research directions. In addition, we have provided cited papers with their corresponding implementations in https://github.com/mindflow-institue/Awesome-Transformer.

Motivation & Objective

• Survey the landscape of Transformer models in medical image analysis across multiple tasks.
• Provide taxonomies and critical analysis of designs, strengths, and limitations.
• Summarize benchmarks, datasets, and practical clinical considerations.
• Identify challenges and propose future research directions.

Proposed method

• Systematic literature review of Transformer-enabled medical imaging papers (over 200 papers).
• Taxonomic categorization of models by task and architectural role (pure vs hybrid Transformers).
• Discussion of datasets, benchmarks, and performance trends across tasks such as classification, segmentation, reconstruction, and detection.
• Analysis of clinical considerations, robustness, privacy, and edge-deployable aspects.

Experimental results

Research questions

• RQ1 What are the main Transformer-based approaches used for each medical image analysis task?
• RQ2 How do pure Transformer and CNN-Transformer hybrid models compare in performance and design trade-offs?
• RQ3 What benchmarks, datasets, and evaluation practices define the current state-of-the-art across tasks?
• RQ4 What open challenges and future directions shape Transformers in medical imaging?

Key findings

• The review covers more than 200 papers in a structured taxonomy.
• ViTs offer advantages in modeling long-range dependencies and provide attention-based interpretability.
• Hybrid CNN-Transformer designs are used to balance global context with local detail.
• There are efforts to enable federated and decentralized training (e.g., FESTA) to address data privacy and scarcity.
• Lightweight and real-time variants (e.g., POCFormer) adapt ViTs for deployment on resource-constrained devices.
• The paper discusses clinical relevance and cites real-world examples like Med-PaLM 2 and SurgicalGPT to illustrate Transformer utility.