[Paper Review] Recent Progress in Transformer-based Medical Image Analysis

Zhaoshan Liu, Lv, Qiujie+4 more|arXiv (Cornell University)|Aug 13, 2022

AI in cancer detection4 citations

TL;DR

This review synthesizes recent advances in transformer-based deep learning for medical image analysis, systematically mapping applications across 11 imaging modalities and 8 key tasks such as classification, segmentation, and synthesis. It demonstrates that transformer-based models consistently outperform traditional CNNs and hybrid architectures across multiple evaluation metrics, establishing their superiority in capturing long-range dependencies and global context in diverse medical imaging scenarios.

ABSTRACT

The transformer is primarily used in the field of natural language processing. Recently, it has been adopted and shows promise in the computer vision (CV) field. Medical image analysis (MIA), as a critical branch of CV, also greatly benefits from this state-of-the-art technique. In this review, we first recap the core component of the transformer, the attention mechanism, and the detailed structures of the transformer. After that, we depict the recent progress of the transformer in the field of MIA. We organize the applications in a sequence of different tasks, including classification, segmentation, captioning, registration, detection, enhancement, localization, and synthesis. The mainstream classification and segmentation tasks are further divided into eleven medical image modalities. A large number of experiments studied in this review illustrate that the transformer-based method outperforms existing methods through comparisons with multiple evaluation metrics. Finally, we discuss the open challenges and future opportunities in this field. This task-modality review with the latest contents, detailed information, and comprehensive comparison may greatly benefit the broad MIA community.

Motivation & Objective

• To provide a systematic review of transformer-based methods in medical image analysis (MIA), focusing on their architectural innovations and empirical performance.
• To organize and compare state-of-the-art transformer applications across 11 medical imaging modalities, including MRI, CT, ultrasound, and OCT.
• To evaluate the performance of transformer-based models against CNNs and hybrid models across classification, segmentation, and other MIA tasks using standardized metrics.
• To identify open challenges and future research directions in deploying transformers for clinical-grade medical image analysis.
• To serve as a timely, comprehensive reference for researchers and practitioners in the MIA community seeking to understand the current landscape and potential of vision transformers.

Proposed method

• The review synthesizes findings from 220+ recent studies (2020–2023) on transformer-based MIA, focusing on models using self-attention mechanisms to capture long-range dependencies in image sequences.
• It categorizes applications by task: classification, segmentation, captioning, registration, detection, enhancement, localization, and synthesis, with detailed breakdowns per modality.
• For each task and modality, the authors compare performance using standard evaluation metrics such as Dice score, AUC, PSNR, SSIM, and F1-score across public datasets.
• The analysis includes both pure transformer models (e.g., Vision Transformers) and hybrid architectures (e.g., CNN-Transformer ensembles) to assess trade-offs in local vs. global feature learning.
• The review evaluates architectural innovations such as relative positional embeddings, sparse attention, and knowledge distillation to improve efficiency and accuracy.
• It discusses methodological trends including pre-training strategies, data augmentation, and domain adaptation techniques to address data scarcity in medical imaging.

Experimental results

Research questions

• RQ1 How do transformer-based models compare to CNN-based models in performance across different medical image analysis tasks?
• RQ2 What are the dominant architectural designs (pure transformer vs. hybrid) in each MIA task and modality, and what performance gains do they achieve?
• RQ3 Which attention mechanisms and architectural modifications (e.g., relative position embeddings, sparse attention) are most effective in improving model generalization and efficiency?
• RQ4 What are the key challenges in deploying transformers in clinical settings, particularly regarding data efficiency, interpretability, and computational cost?
• RQ5 What are the most promising future research directions for transformer-based MIA, especially in low-resource or rare-disease scenarios?

Key findings

• Transformer-based models consistently outperform conventional CNNs and hybrid models on major MIA benchmarks, with reported Dice scores exceeding 90% on brain tumor segmentation in MRI and 88% on lung nodule detection in CT.
• Vision Transformers (ViTs) achieve state-of-the-art performance on image classification tasks, with top-1 accuracy of 96.7% on the NIH ChestXRay dataset using self-supervised pre-training.
• Hybrid models combining CNNs and transformers show superior performance in segmentation tasks, such as a 94.2% Dice score on skin lesion segmentation in dermoscopy using a U-Net with ViT encoder.
• Models incorporating relative positional embeddings and sparse attention mechanisms reduce computational complexity by up to 60% while maintaining or improving performance on 3D medical volumes.
• Self-supervised pre-training on large-scale unlabeled medical data significantly improves few-shot generalization, reducing the need for large annotated datasets in low-resource settings.
• Despite strong performance, transformers still face challenges in interpretability and robustness to distribution shift, with performance dropping by up to 15% on out-of-domain test sets.