[Paper Review] A ChatGPT Aided Explainable Framework for Zero-Shot Medical Image Diagnosis

Jiaxiang Liu, Tianxiang Hu+4 more|arXiv (Cornell University)|Jul 5, 2023

COVID-19 diagnosis using AI14 citations

TL;DR

The paper proposes a CLIP-based zero-shot medical image classification framework augmented with ChatGPT-generated visual symptom prompts to improve accuracy and explainability without training, evaluated on five medical datasets.

ABSTRACT

Zero-shot medical image classification is a critical process in real-world scenarios where we have limited access to all possible diseases or large-scale annotated data. It involves computing similarity scores between a query medical image and possible disease categories to determine the diagnostic result. Recent advances in pretrained vision-language models (VLMs) such as CLIP have shown great performance for zero-shot natural image recognition and exhibit benefits in medical applications. However, an explainable zero-shot medical image recognition framework with promising performance is yet under development. In this paper, we propose a novel CLIP-based zero-shot medical image classification framework supplemented with ChatGPT for explainable diagnosis, mimicking the diagnostic process performed by human experts. The key idea is to query large language models (LLMs) with category names to automatically generate additional cues and knowledge, such as disease symptoms or descriptions other than a single category name, to help provide more accurate and explainable diagnosis in CLIP. We further design specific prompts to enhance the quality of generated texts by ChatGPT that describe visual medical features. Extensive results on one private dataset and four public datasets along with detailed analysis demonstrate the effectiveness and explainability of our training-free zero-shot diagnosis pipeline, corroborating the great potential of VLMs and LLMs for medical applications.

Motivation & Objective

• Demonstrate the feasibility of integrating a large language model (ChatGPT) with a vision-language model (CLIP) to improve zero-shot medical image diagnosis.
• Improve explainability by generating disease-specific visual symptoms to guide CLIP-based classification.
• Evaluate the proposed framework across diverse medical datasets to assess accuracy gains and interpretability.
• Investigate prompt design and aggregation strategies that impact performance and explanations.

Proposed method

• Use CLIP to extract visual features from medical images and compute similarity with text representations.
• Query ChatGPT with designed prompts to generate disease-specific visual symptoms describing useful features for diagnosis.
• Encode ChatGPT-generated symptom texts with CLIP's text encoder and aggregate similarity scores across symptoms to obtain a final score per disease.
• Compute the final diagnosis by selecting the category with the highest averaged similarity between image features and symptom texts.
• Design prompts to emphasize medically relevant features and grounding in literature to improve description quality and reduce hallucinations.
• Compare aggregation strategies (mean vs max) and prompt designs, and contrast with OpenFlamingo as a baseline.

Experimental results

Research questions

• RQ1 Can ChatGPT-generated disease symptoms improve zero-shot medical image classification when used with CLIP?
• RQ2 How does prompt design affect the quality of explanations and the diagnostic accuracy in a CLIP-LLM pipeline?
• RQ3 What aggregation strategy (mean vs max) yields better zero-shot diagnostic performance across medical datasets?
• RQ4 How does the proposed method compare to open multimodal models like OpenFlamingo in medical diagnosis tasks?

Key findings

• The method yields consistent improvements over standard CLIP zero-shot classification across all five datasets.
• On Pneumonia, the approach improves accuracy by up to 11.73 percentage points compared with CLIP using only category names.
• On Shenzhen, accuracy improves by up to 17.37 percentage points.
• Compared with OpenFlamingo, the framework generally outperforms it by 2.59% to 5.80% on most datasets, with BrainTumor showing a 5.59% gain.
• With the designed prompt, four of five datasets show improved accuracy over the baseline prompt, demonstrating the impact of prompt engineering.
• The approach provides interpretable attention maps showing alignment between image regions and ChatGPT-generated symptoms, supporting explainability.