[Paper Review] Artificial General Intelligence for Medical Imaging Analysis

Xiang Li, Lin Zhao+17 more|arXiv (Cornell University)|Jun 8, 2023

Radiomics and Machine Learning in Medical Imaging12 citations

TL;DR

This is a comprehensive review of how Artificial General Intelligence (AGI) models, especially LLMs and multimodal models, can be adapted for medical imaging and healthcare, outlining roadmaps, applications, challenges, and domain-specific considerations.

ABSTRACT

Large-scale Artificial General Intelligence (AGI) models, including Large Language Models (LLMs) such as ChatGPT/GPT-4, have achieved unprecedented success in a variety of general domain tasks. Yet, when applied directly to specialized domains like medical imaging, which require in-depth expertise, these models face notable challenges arising from the medical field's inherent complexities and unique characteristics. In this review, we delve into the potential applications of AGI models in medical imaging and healthcare, with a primary focus on LLMs, Large Vision Models, and Large Multimodal Models. We provide a thorough overview of the key features and enabling techniques of LLMs and AGI, and further examine the roadmaps guiding the evolution and implementation of AGI models in the medical sector, summarizing their present applications, potentialities, and associated challenges. In addition, we highlight potential future research directions, offering a holistic view on upcoming ventures. This comprehensive review aims to offer insights into the future implications of AGI in medical imaging, healthcare, and beyond.

Motivation & Objective

• Assess how foundational AGI models can be tailored for medical imaging and healthcare while respecting privacy and regulatory constraints.
• Identify data, knowledge, and modeling strategies to integrate expert knowledge, multimodal data, and domain-specific prompts.
• Outline practical roadmaps for deploying AGI in clinical workflows and education.
• Highlight potential applications and critical challenges in real-world medical settings.

Proposed method

• Survey the characteristics and technological foundations of LLMs/AGI (emergent abilities, multimodal learning, transformers, in-context learning, prompt engineering, RLHF).
• Discuss expert-in-the-loop strategies and domain tailoring to medical imaging and healthcare tasks.
• Propose multimodal modeling and integration practices (text, images, and clinical data) and the role of knowledge graphs and medical informatics.
• Examine data privacy, regulation, collaboration, and deployment considerations for hospital-scale AGI systems.

Figure 1: Illustration of a) in-context learning and b) prompt engineering.

Experimental results

Research questions

• RQ1 How can LLMs and multimodal AGI models be adapted to the unique demands of medical imaging and healthcare?
• RQ2 What data, knowledge, and modeling strategies are needed to effectively tailor AGI to medical domains while preserving privacy and compliance?
• RQ3 What roadmaps and practical steps are proposed for deploying AGI tools in clinical workflows, education, and research?
• RQ4 What are the major challenges and potential pitfalls in using AGI for medical imaging and healthcare, and how might they be mitigated?

Key findings

• The paper provides a structured set of design principles and roadmaps for adapting AGI to medical imaging, including data, knowledge, and model considerations.
• It discusses expert-in-the-loop, domain tailoring, prompt tuning, multimodal prompts, and RLHF as key enablers for healthcare AGI.
• The review highlights practical applications such as disease diagnosis, patient outcome prediction, medical education, and streamlined clinical documentation, while noting regulatory, privacy, data availability, and deployment challenges.
• No quantitative results are reported in this review; the insights are qualitative assessments of capabilities, challenges, and deployment considerations.