[论文解读] Is Open-Source There Yet? A Comparative Study on Commercial and Open-Source LLMs in Their Ability to Label Chest X-Ray Reports

Felix J. Dorfner, Liv Jürgensen+12 more|arXiv (Cornell University)|Feb 19, 2024

Radiomics and Machine Learning in Medical Imaging被引用 4

一句话总结

本研究比较了商用（GPT-3.5 Turbo、GPT-4）与开源大语言模型（Mistral-7B、Mixtral-8x7B、Llama2-13B/70B、QWEN1.5-72B、CheXbert、CheXpert-labeler）在使用零样本提示和少样本提示对胸部X光报告中的发现进行标注时的表现。开源模型，尤其是采用少样本提示或集成方法时，性能几乎与GPT-4相当，为放射科报告分类提供了成本更低、保护隐私且可复现的替代方案。

ABSTRACT

Introduction: With the rapid advances in large language models (LLMs), there have been numerous new open source as well as commercial models. While recent publications have explored GPT-4 in its application to extracting information of interest from radiology reports, there has not been a real-world comparison of GPT-4 to different leading open-source models. Materials and Methods: Two different and independent datasets were used. The first dataset consists of 540 chest x-ray reports that were created at the Massachusetts General Hospital between July 2019 and July 2021. The second dataset consists of 500 chest x-ray reports from the ImaGenome dataset. We then compared the commercial models GPT-3.5 Turbo and GPT-4 from OpenAI to the open-source models Mistral-7B, Mixtral-8x7B, Llama2-13B, Llama2-70B, QWEN1.5-72B and CheXbert and CheXpert-labeler in their ability to accurately label the presence of multiple findings in x-ray text reports using different prompting techniques. Results: On the ImaGenome dataset, the best performing open-source model was Llama2-70B with micro F1-scores of 0.972 and 0.970 for zero- and few-shot prompts, respectively. GPT-4 achieved micro F1-scores of 0.975 and 0.984, respectively. On the institutional dataset, the best performing open-source model was QWEN1.5-72B with micro F1-scores of 0.952 and 0.965 for zero- and few-shot prompting, respectively. GPT-4 achieved micro F1-scores of 0.975 and 0.973, respectively. Conclusion: In this paper, we show that while GPT-4 is superior to open-source models in zero-shot report labeling, the implementation of few-shot prompting can bring open-source models on par with GPT-4. This shows that open-source models could be a performant and privacy preserving alternative to GPT-4 for the task of radiology report classification.

研究动机与目标

• 评估开源大语言模型是否能在标注胸部X光报告中的发现方面达到与GPT-4等商用模型相当的性能。
• 评估提示策略（零样本与少样本）在不同数据集上对模型性能的影响。
• 探究将开源大语言模型作为临床自然语言处理中专有模型的隐私保护、成本效益高且可复现的替代方案的可行性。
• 探索结合表现最佳的开源大语言模型的集成方法，以提高分类准确性。
• 解决医学报告标注任务中类别不平衡与提示优化带来的挑战。

提出的方法

• 使用了两个独立的数据集：来自马萨诸塞州总医院的540份机构性胸部X光报告和来自ImaGenome数据集的500份报告。
• 评估了六种开源大语言模型（Mistral-7B、Mixtral-8x7B、Llama2-13B、Llama2-70B、QWEN1.5-72B、CheXbert、CheXpert-labeler）以及两种商用模型（GPT-3.5 Turbo、GPT-4）。
• 应用了零样本提示和少样本提示技术，提示在初始阶段于GPT-4上设计并测试，随后适配至其他模型。
• 通过14种放射学发现的微平均和宏平均F1分数衡量性能，机构数据集中通过过采样处理类别不平衡问题。
• 通过多数投票法将表现最佳的三种开源模型（Mixtral-8x7B、Llama2-70B、QWEN1.5-72B）的预测结果组合，构建集成模型。
• 模型推理在本地或云服务提供商上进行，量化技术使模型能够在硬件资源有限的设备上部署（例如，Llama-2 70B在Int4下仅需48 GB显存）

Figure 1: Flow-chart for the reports from the ImaGenome and the institutional dataset

实验结果

研究问题

• RQ1 开源大语言模型是否能在未结构化胸部X光报告的发现分类任务中实现与GPT-4相当的性能？
• RQ2 与零样本提示相比，少样本提示在提升开源大语言模型此任务准确率方面表现如何？
• RQ3 通过组合多个开源模型的集成方法，能在多大程度上缩小与GPT-4的性能差距？
• RQ4 在医疗人工智能中，开源大语言模型相较于商用模型在成本、隐私和可复现性方面具有哪些实际优势？
• RQ5 类别不平衡与提示设计对放射科报告分类任务中模型性能有何影响？

主要发现

• 在ImaGenome数据集中，表现最佳的开源模型Llama2-70B在零样本提示下的微平均F1得分为0.972，少样本提示下为0.970，而GPT-4分别为0.975和0.984。
• 在机构数据集中，QWEN1.5-72B在零样本提示下的微平均F1得分为0.952，少样本提示下为0.965，而GPT-4分别为0.975和0.973。
• 在机构数据集中，Mixtral-8x7B、Llama2-70B和QWEN1.5-72B的集成模型在少样本提示下与GPT-4性能相当，微平均F1得分分别为0.971和0.973。
• 该集成模型在两个数据集的零样本和少样本任务中均优于GPT-3.5 Turbo，证明了模型组合的价值。
• 开源模型在“气胸”和“肺不张”等常被报告中直接提及的发现上表现优异，标签常以字面形式出现。
• 本研究证实，开源大语言模型在成本、隐私和可复现性方面具有显著优势，尤其当模型在本地部署且不依赖外部API时。

Figure 2: Outline of the zero-shot prompting workflow for the ImaGenome dataset. The model is given instructions and the report to classify and outputs its response according to the JSON formatted template with the corresponding labels.