[论文解读] Evaluating LLM -- Generated Multimodal Diagnosis from Medical Images and Symptom Analysis

Dimitrios P. Panagoulias, Maria Virvou+1 more|arXiv (Cornell University)|Jan 28, 2024

Radiomics and Machine Learning in Medical Imaging被引用 11

一句话总结

本文提出一个两步式LLM评估范式，结合多模态交互与领域特定分析，评估GPT-4-Vision-Preview在带图像的病理MCQs上的表现，准确率约为84%，并通过NER和知识图谱提取洞见。

ABSTRACT

Large language models (LLMs) constitute a breakthrough state-of-the-art Artificial Intelligence technology which is rapidly evolving and promises to aid in medical diagnosis. However, the correctness and the accuracy of their returns has not yet been properly evaluated. In this work, we propose an LLM evaluation paradigm that incorporates two independent steps of a novel methodology, namely (1) multimodal LLM evaluation via structured interactions and (2) follow-up, domain-specific analysis based on data extracted via the previous interactions. Using this paradigm, (1) we evaluate the correctness and accuracy of LLM-generated medical diagnosis with publicly available multimodal multiple-choice questions(MCQs) in the domain of Pathology and (2) proceed to a systemic and comprehensive analysis of extracted results. We used GPT-4-Vision-Preview as the LLM to respond to complex, medical questions consisting of both images and text, and we explored a wide range of diseases, conditions, chemical compounds, and related entity types that are included in the vast knowledge domain of Pathology. GPT-4-Vision-Preview performed quite well, scoring approximately 84\% of correct diagnoses. Next, we further analyzed the findings of our work, following an analytical approach which included Image Metadata Analysis, Named Entity Recognition and Knowledge Graphs. Weaknesses of GPT-4-Vision-Preview were revealed on specific knowledge paths, leading to a further understanding of its shortcomings in specific areas. Our methodology and findings are not limited to the use of GPT-4-Vision-Preview, but a similar approach can be followed to evaluate the usefulness and accuracy of other LLMs and, thus, improve their use with further optimization.

研究动机与目标

• 提出一种在医学领域中多模态LLM的两步评估范式（多模态评估与领域特定分析）。
• 使用带图像+文本的MCQs评估LLM生成的病理诊断的正确性和准确性。
• 结合IMA、NER和知识图谱分析结果，识别弱点并指导微调。
• 在公开可用的病理MCQs上演示使用GPT-4-Vision-Preview的方法。

提出的方法

• 使用带预定义行为准则的图像+文本MCQs进行结构化的多模态交互。
• 提示工程以强制特定的回答格式和简洁的答案。
• 数据提取包括 Image Metadata Analysis (IMA)、Named Entity Recognition (NER) 和 Knowledge Graphs (KGs)。
• 领域特定分析以从正确与错误的回答和解释中导出微调需求。

Figure 1: Multimodal LLM evaluation

实验结果

研究问题

• RQ1 在将医学影像与基于症状的文本结合时，LLMs诊断的准确性有多高？
• RQ2 通过 IMA、NER 与 KG 分析，LLM揭示了哪些弱点或知识路径？
• RQ3 该评估方法能否为领域特定的医疗任务提供针对性的微调或再训练指南？
• RQ4 该方法是否可推广到除 GPT-4-Vision-Preview 之外的其他LLM？

主要发现

• GPT-4-Vision-Preview 在多模态病理MCQs上的诊断正确率约为84%。
• 正确答案：66/79；错误答案：13/79，覆盖79道带图像的问题。
• 病理子领域表现差异较大，例如一般病理：动脉粥样硬化与血栓形成 8/10，细胞损伤 7/10，免疫病理学 9/10，炎症 6/10，肿瘤学 10/10；器官系统病理：心血管 9/9，皮肤病理解剖 9/10，内分泌 8/10。
• IMA 识别出的图像领域弱点主要在心血管、皮肤和内分泌图像。
• NER 与 KG 分析揭示了特定实体及知识路径的薄弱点，可在微调中进行针对性改进。

Figure 2: Domain specific Analysis