[Paper Review] GPT-4 Vision on Medical Image Classification -- A Case Study on COVID-19 Dataset

Ruibo Chen, Tianyi Xiong+7 more|arXiv (Cornell University)|Oct 27, 2023

COVID-19 diagnosis using AI4 citations

TL;DR

This study investigates the application of GPT-4V, a vision-language foundation model, to medical image classification using in-context learning on a COVID-19 X-ray dataset. By designing task-specific prompts, GPT-4V achieves classification accuracy comparable to fine-tuned CNNs like ResNet-18 and VGG-16, even in few-shot settings, demonstrating the potential of large models to bypass traditional data-hungry training paradigms in medical imaging.

ABSTRACT

This technical report delves into the application of GPT-4 Vision (GPT-4V) in the nuanced realm of COVID-19 image classification, leveraging the transformative potential of in-context learning to enhance diagnostic processes.

Motivation & Objective

• To explore the feasibility of using GPT-4V, a large vision-language foundation model, for medical image classification without fine-tuning.
• To evaluate whether in-context learning with carefully designed prompts can enable GPT-4V to match or exceed performance of conventional deep learning models in few-shot settings.
• To investigate the impact of prompt engineering, including reasoning explanations, on GPT-4V’s classification performance in medical imaging.
• To compare GPT-4V’s performance against established CNN-based models (ResNet-18, VGG-16) on a real-world COVID-19 X-ray dataset under both full and few-shot training regimes.

Proposed method

• The study employs in-context learning (ICL) to guide GPT-4V using prompt templates that include input-output demonstrations for image classification.
• Multiple prompt variants (ICL1 to ICL4, ICL-R1, ICL-R2) were designed with varying degrees of task specificity, structure, and inclusion of reasoning rationale.
• Prompts were constructed to embed clinical context and classification rules, leveraging GPT-4V’s pre-trained knowledge for zero- or few-shot adaptation.
• Images were processed either individually or consolidated into a single input image to assess the model’s attention and performance consistency.
• Baseline models (ResNet-18 and VGG-16) were fine-tuned on the full dataset and on a 6-shot subset to enable fair comparison.
• Evaluation metrics included precision, recall, F1-score, and overall accuracy on the test set of 46 X-ray images (26 COVID, 20 normal).

Experimental results

Research questions

• RQ1 Can GPT-4V achieve competitive classification accuracy on medical images using only in-context learning, without any fine-tuning?
• RQ2 How does prompt design—particularly structure, inclusion of reasoning, and image consolidation—affect GPT-4V’s performance in few-shot medical image classification?
• RQ3 Does the integration of reasoning explanations in prompts improve GPT-4V’s classification accuracy on medical X-rays?
• RQ4 How does GPT-4V’s performance compare to fine-tuned CNNs (ResNet-18, VGG-16) under both full and few-shot training conditions?

Key findings

• GPT-4V with the ICL4 prompt achieved an overall accuracy of 85%, outperforming the 6-shot fine-tuned ResNet-18 (74%) and VGG-16 (80%) baselines.
• The ICL4 prompt variant achieved a balanced F1-score of 0.85 for COVID and 0.84 for normal cases, indicating strong generalization across both classes.
• Consolidating all images into a single input improved performance compared to individual image processing, suggesting better attention distribution by GPT-4V.
• Adding reasoning explanations to prompts (ICL-R1, ICL-R2) did not improve performance and sometimes degraded results, indicating potential misalignment between provided rationale and model comprehension.
• GPT-4V with ICL4 achieved 95% precision and 95% recall for the normal class, outperforming the full-fine-tuned ResNet-18 (95% precision, 90% recall) and matching the VGG-16 baseline (100% precision, 92% recall).
• The model’s performance was consistent across multiple random seeds, with average results reported over five runs, indicating robustness in the few-shot setting.