[Paper Review] Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4V

Zhiling Yan, Kai Zhang+4 more|arXiv (Cornell University)|Oct 29, 2023

Artificial Intelligence in Healthcare and Education33 citations

TL;DR

The paper evaluates GPT-4V on medical visual question answering across pathology and radiology datasets from 11 modalities and 15 objects, finding current GPT-4V unreliable for real-world diagnostics and identifying seven behavioral facets.

ABSTRACT

In this paper, we critically evaluate the capabilities of the state-of-the-art multimodal large language model, i.e., GPT-4 with Vision (GPT-4V), on Visual Question Answering (VQA) task. Our experiments thoroughly assess GPT-4V's proficiency in answering questions paired with images using both pathology and radiology datasets from 11 modalities (e.g. Microscopy, Dermoscopy, X-ray, CT, etc.) and fifteen objects of interests (brain, liver, lung, etc.). Our datasets encompass a comprehensive range of medical inquiries, including sixteen distinct question types. Throughout our evaluations, we devised textual prompts for GPT-4V, directing it to synergize visual and textual information. The experiments with accuracy score conclude that the current version of GPT-4V is not recommended for real-world diagnostics due to its unreliable and suboptimal accuracy in responding to diagnostic medical questions. In addition, we delineate seven unique facets of GPT-4V's behavior in medical VQA, highlighting its constraints within this complex arena. The complete details of our evaluation cases are accessible at https://github.com/ZhilingYan/GPT4V-Medical-Report.

Motivation & Objective

• Assess GPT-4V's ability to answer visually paired medical questions across multiple imaging modalities and organs.
• Characterize GPT-4V's localization, size estimation, and integration of image and text for medical queries.
• Identify strengths, limitations, and behavioral patterns of GPT-4V in a medical VQA setting.
• Provide guidance on when and how GPT-4V may assist or mislead in medical contexts.

Proposed method

• Use zero-shot prompts to query GPT-4V via the ChatGPT interface with image inputs across cases from PathVQA, VQA-RAD, and PMC-VQA.
• Evaluate accuracy on closed-ended (multiple choice) and open-ended questions using ground-truth labels.
• Curate a dataset of 133 samples across 11 modalities and 15 objects, with 16 question types spanning pathology and radiology.
• Analyze performance across difficulty levels (easy, medium, hard) and specific question types (modality recognition, localization, size, etc.).
• Identify seven behavioral facets of GPT-4V behavior in medical VQA, including reliance on cues, size assessment challenges, and textual bias.

Experimental results

Research questions

• RQ1 Can GPT-4V recognize medical imaging modalities and localize objects across pathology and radiology images?
• RQ2 How accurate is GPT-4V for medical VQA in zero-shot settings, and how does it vary by question type and difficulty?
• RQ3 What are the main limitations and behavioral patterns of GPT-4V when applied to medical VQA?
• RQ4 Is GPT-4V suitable for real-world diagnostic support based on its VQA performance?

Key findings

• Overall pathology VQA accuracy is 29.9%, with 35.3% on closed-ended questions.
• Radiology VQA achieves 50.0% overall accuracy, with 81.25% easy, 59.09% medium, and 11.11% hard questions.
• GPT-4V shows better performance on easy questions but struggles with harder, multi-slice, and size-estimation tasks.
• GPT-4V often relies on textual context over visual data, leading to biased or incomplete interpretations.
• The model tends to be cautious, frequently stating it is not a medical professional and providing long explanations.
• The study concludes that GPT-4V is not recommended for real-world diagnostic use due to unreliable accuracy.
• Seven distinct behavioral facets of GPT-4V in medical VQA are identified and discussed.