[论文解读] Holistic Evaluation of GPT-4V for Biomedical Imaging

Zhengliang Liu, Hanqi Jiang+48 more|arXiv (Cornell University)|Nov 10, 2023

Artificial Intelligence in Healthcare and Education被引用 12

一句话总结

本文对 GPT-4V 在生物医学成像任务上的能力进行了大规模、跨领域的评估，涵盖 16 种成像领域的模态识别、定位、诊断和报告生成。它确定了在模态/解剖结构识别和图像描述方面的优势，以及在疾病诊断和精确定位方面的局限性。

ABSTRACT

In this paper, we present a large-scale evaluation probing GPT-4V's capabilities and limitations for biomedical image analysis. GPT-4V represents a breakthrough in artificial general intelligence (AGI) for computer vision, with applications in the biomedical domain. We assess GPT-4V's performance across 16 medical imaging categories, including radiology, oncology, ophthalmology, pathology, and more. Tasks include modality recognition, anatomy localization, disease diagnosis, report generation, and lesion detection. The extensive experiments provide insights into GPT-4V's strengths and weaknesses. Results show GPT-4V's proficiency in modality and anatomy recognition but difficulty with disease diagnosis and localization. GPT-4V excels at diagnostic report generation, indicating strong image captioning skills. While promising for biomedical imaging AI, GPT-4V requires further enhancement and validation before clinical deployment. We emphasize responsible development and testing for trustworthy integration of biomedical AGI. This rigorous evaluation of GPT-4V on diverse medical images advances understanding of multimodal large language models (LLMs) and guides future work toward impactful healthcare applications.

研究动机与目标

• Evaluate GPT-4V across diverse biomedical imaging modalities (e.g., X-ray, MRI, CT, microscopy) to assess modality recognition capabilities.
• Assess GPT-4V’s ability to localize anatomical structures within biomedical images.
• Benchmark GPT-4V’s image classification/diagnosis performance on biomedical tasks.
• Test GPT-4V’s capacity to generate diagnostic-style reports from medical images.
• Provide insights into strengths, limitations, and implications for responsible, clinical deployment of biomedical AGI.

提出的方法

• Utilize GPT-4V to perform zero-shot and direct classification on multiple public chest radiography datasets (e.g., MIMIC-CXR, CheXpert, ChestXray2017, COVID-Qu-Ex, OpenI, SIIM-ACR, NIH Chest X-rays).
• Evaluate GPT-4V’s ability to output multi-class labels and to generate structured diagnostic reports with findings and impressions.
• Extend evaluation to neuroimaging, oncological imaging for radiotherapy planning, cytopathology, ophthalmology, medical robotics, neurological disease imaging, biological imaging, cardiac imaging, ultrasound, nuclear medicine, endoscopy, dermatology, genetics, orthopedic/pediatric, and dental imaging.
• In neuroimaging, test on high-resolution datasets (e.g., R1741 mouse brain, Allen Brain Institute atlas) to assess knowledge transfer and reconstruction quality evaluation.
• In oncological imaging, use TCIA datasets (Burdenko-GBM-Progression, GLIS-RT, Lung-PET-CT-Dx) and breast X-ray (DDSM) for multimodal assessment.
• For cytopathology, employ LC25000 and ALL datasets to evaluate cellular-level diagnostic capabilities, including limitations in staging.

实验结果

研究问题

• RQ1 Can GPT-4V reliably recognize imaging modalities and anatomical regions across diverse biomedical datasets?
• RQ2 How well does GPT-4V perform disease diagnosis and lesion localization in biomedical images?
• RQ3 What is GPT-4V’s ability to generate coherent and clinically useful radiology or pathology reports from images?
• RQ4 What are the strengths and limitations of GPT-4V when applied to non-radiology biomedical imaging tasks (e.g., ophthalmology, pathology, neuroscience, biology, robotics)?
• RQ5 What considerations are necessary for responsible validation and deployment of biomedical multimodal LLMs like GPT-4V in clinical or research settings?

主要发现

• GPT-4V 在跨多种成像领域的模态识别和解剖定位方面表现出熟练度。
• GPT-4V 在诊断报告生成方面表现出色，表明具有强大的图像描述能力。
• GPT-4V 在某些任务中在疾病诊断和病变的精确定位方面存在困难。
• 通过额外的上下文提示和领域信息可以看到性能提升，但在高风险临床决策方面仍存在差距。
• 研究强调在生物医学 AGI 部署中需要进行严格的验证、偏差评估以及人机协同监管。