[Paper Review] Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis

Chaoyi Wu, Jiayu Lei+9 more|arXiv (Cornell University)|Oct 15, 2023

Artificial Intelligence in Healthcare and Education46 citations

TL;DR

This study assesses GPT-4V’s capabilities across multimodal medical diagnosis over 17 body systems and 8 imaging modalities, highlighting strengths in modality/anatomy recognition but major gaps in diagnosis, reporting, localization, and multi-image reasoning.

ABSTRACT

Driven by the large foundation models, the development of artificial intelligence has witnessed tremendous progress lately, leading to a surge of general interest from the public. In this study, we aim to assess the performance of OpenAI's newest model, GPT-4V(ision), specifically in the realm of multimodal medical diagnosis. Our evaluation encompasses 17 human body systems, including Central Nervous System, Head and Neck, Cardiac, Chest, Hematology, Hepatobiliary, Gastrointestinal, Urogenital, Gynecology, Obstetrics, Breast, Musculoskeletal, Spine, Vascular, Oncology, Trauma, Pediatrics, with images taken from 8 modalities used in daily clinic routine, e.g., X-ray, Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), Digital Subtraction Angiography (DSA), Mammography, Ultrasound, and Pathology. We probe the GPT-4V's ability on multiple clinical tasks with or without patent history provided, including imaging modality and anatomy recognition, disease diagnosis, report generation, disease localisation. Our observation shows that, while GPT-4V demonstrates proficiency in distinguishing between medical image modalities and anatomy, it faces significant challenges in disease diagnosis and generating comprehensive reports. These findings underscore that while large multimodal models have made significant advancements in computer vision and natural language processing, it remains far from being used to effectively support real-world medical applications and clinical decision-making. All images used in this report can be found in https://github.com/chaoyi-wu/GPT-4V_Medical_Evaluation.

Motivation & Objective

• Assess GPT-4V's ability to recognize medical image modalities and anatomy.
• Evaluate GPT-4V's performance on diagnosis, report generation, and localization across multiple imaging modalities.
• Examine effects of patient history and multi-image inputs on GPT-4V outputs.
• Identify limitations and safety considerations for clinical use of GPT-4V in radiology and pathology.

Proposed method

• Select case studies from Radiopaedia for radiology across 17 body systems and 8 imaging modalities.
• Input up to four 2D images into GPT-4V via the online interface and prompt for tasks including diagnosis, report generation, and localization.
• Use reference annotations from Radiopaedia/PathologyOutlines as a correctness baseline while noting limitations in standard clinical formatting.
• Conduct two-round conversations for pathology evaluation (image-only then image plus tissue origin) and stepwise localization tasks (presence, bounding box, IOU).
• Clamp and normalize image intensities and select key slices consistent with expert radiologist guidance; evaluate multi-image inputs and cross-modal inputs separately.

Figure 1 : The Diagram of Medical Systems and Imaging Modalities. In this paper we comprehensively consider 17 medical systems (Figure a ) and our cases can cover 8 different imaging modalities (Figure b ), i.e. , X-ray, CT, MRI, PET, DSA, Mammography, Ultrasound, Pathology from left to right.

Experimental results

Research questions

• RQ1 Can GPT-4V correctly recognize imaging modality and anatomical structures in medical images?
• RQ2 Can GPT-4V localize anatomical structures or anomalies within medical images?
• RQ3 Can GPT-4V generate accurate and clinically meaningful radiology or pathology reports?
• RQ4 How does GPT-4V perform when integrating patient history or multiple images from different modalities?
• RQ5 What are the limitations and safety considerations for using GPT-4V in real-world medical decision making?

Key findings

• GPT-4V can recognize imaging modality and anatomy across many cases.
• GPT-4V struggles with accurate disease diagnosis and comprehensive report generation.
• GPT-4V can generate structured reports but content is often incorrect.
• GPT-4V can OCR text and markers in images but may misinterpret annotations.
• GPT-4V can identify medical devices and their locations.
• GPT-4V has difficulty analyzing multiple images and maintaining context across rounds.
• GPT-4V predictions are heavily influenced by patient medical history.
• GPT-4V cannot reliably localize anatomical structures or anomalies (low IOU, high variance).
• Performance varies and shows inconsistency and safety concerns in outputs.

Figure 2 : A Demonstration Case From Central Nervous System . “Cont.” denotes this sample is a continuation of the case titled as “Central Nervous System: Case 3”. Red in the figure denotes the incorrect parts, green denoting correct parts and yellow for uncertainty. The colored sections within the