[Paper Review] Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions

Kai Sun, S. D. Xue+13 more|arXiv (Cornell University)|Dec 3, 2024

Biomedical Text Mining and Ontologies5 citations

TL;DR

This review analyzes Medical Multimodal Foundation Models (MMFMs), detailing datasets, architectures (MMVFMs and MMVLFMs), clinical applications, challenges, and future directions.

ABSTRACT

Recent advancements in deep learning have significantly revolutionized the field of clinical diagnosis and treatment, offering novel approaches to improve diagnostic precision and treatment efficacy across diverse clinical domains, thus driving the pursuit of precision medicine. The growing availability of multi-organ and multimodal datasets has accelerated the development of large-scale Medical Multimodal Foundation Models (MMFMs). These models, known for their strong generalization capabilities and rich representational power, are increasingly being adapted to address a wide range of clinical tasks, from early diagnosis to personalized treatment strategies. This review offers a comprehensive analysis of recent developments in MMFMs, focusing on three key aspects: datasets, model architectures, and clinical applications. We also explore the challenges and opportunities in optimizing multimodal representations and discuss how these advancements are shaping the future of healthcare by enabling improved patient outcomes and more efficient clinical workflows.

Motivation & Objective

• Assess how MMFMs integrate multimodal medical data to improve diagnostic and treatment outcomes.
• Survey datasets, architectures, and clinical tasks enabled by MMFMs across organ systems.
• Identify current challenges in multimodal representation learning and clinical deployment.
• Discuss opportunities and future directions for advancing MMFMs toward precision medicine.

Proposed method

• Classify MMFM variants into Medical Multimodal Vision Foundation Models (MMVFMs) and Medical Multimodal Vision-Language Foundation Models (MMVLFMs).
• Analyze training workflows using diverse multimodal datasets to enable downstream tasks such as segmentation, classification, detection, registration, and report generation.
• Review foundational concepts including transformers, self-supervised learning, and vision transformers (ViT) as they pertain to medical multimodal data.
• Summarize large-scale medical datasets by modality (text, image, image-text) and their roles in pretraining MMFMs.
• Discuss architectural strategies for fusing multimodal information, including input-level fusion versus latent-space fusion.

Experimental results

Research questions

• RQ1 What are the main architectural categories and training paradigms used for MMFMs in medicine?
• RQ2 How do multimodal datasets (text, image, image-text) influence model generalization and clinical task performance?
• RQ3 What clinical applications have MMFMs been applied to, and what challenges limit their deployment?
• RQ4 What future directions are proposed to advance MMFMs toward more robust, generalizable, and clinically useful systems.

Key findings

• MMFMs show potential to unify multimodal medical data for broader diagnostic and treatment tasks.
• Two main fusion paradigms exist: early-stage input fusion and late-stage latent-space fusion, each with trade-offs.
• Large-scale multimodal medical datasets across text, image, and image-text modalities are essential for robust pretraining.
• Applications span segmentation, classification, diagnosis, and report generation, with emphasis on cardiovascular and imaging domains.
• The review identifies challenges in data heterogeneity, annotation scarcity, and clinical workflow integration, and outlines future directions for improving generalization and clinical impact.