[论文解读] Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data

Felix J. Dorfner, Amin Dada+9 more|arXiv (Cornell University)|Aug 25, 2024

Cancer Genomics and Diagnostics被引用 5

一句话总结

生物医学大模型在领域数据上微调后，在多项临床任务的未见医学数据上普遍不及通用模型，表明生物医学微调价值有限，并凸显检索增强方法作为有前景的替代方案。

ABSTRACT

Large language models (LLMs) have shown potential in biomedical applications, leading to efforts to fine-tune them on domain-specific data. However, the effectiveness of this approach remains unclear. This study evaluates the performance of biomedically fine-tuned LLMs against their general-purpose counterparts on a variety of clinical tasks. We evaluated their performance on clinical case challenges from the New England Journal of Medicine (NEJM) and the Journal of the American Medical Association (JAMA) and on several clinical tasks (e.g., information extraction, document summarization, and clinical coding). Using benchmarks specifically chosen to be likely outside the fine-tuning datasets of biomedical models, we found that biomedical LLMs mostly perform inferior to their general-purpose counterparts, especially on tasks not focused on medical knowledge. While larger models showed similar performance on case tasks (e.g., OpenBioLLM-70B: 66.4% vs. Llama-3-70B-Instruct: 65% on JAMA cases), smaller biomedical models showed more pronounced underperformance (e.g., OpenBioLLM-8B: 30% vs. Llama-3-8B-Instruct: 64.3% on NEJM cases). Similar trends were observed across the CLUE (Clinical Language Understanding Evaluation) benchmark tasks, with general-purpose models often performing better on text generation, question answering, and coding tasks. Our results suggest that fine-tuning LLMs to biomedical data may not provide the expected benefits and may potentially lead to reduced performance, challenging prevailing assumptions about domain-specific adaptation of LLMs and highlighting the need for more rigorous evaluation frameworks in healthcare AI. Alternative approaches, such as retrieval-augmented generation, may be more effective in enhancing the biomedical capabilities of LLMs without compromising their general knowledge.

研究动机与目标

• 评估生物医学微调是否能在未见的临床数据和任务上提升大语言模型的性能。
• 在多样化的临床基准上，将生物医学微调的LLMs与通用基线进行对比。
• 研究在医疗保健AI中，领域自适应在哪些任务领域可能有益、在哪些任务中可能没有收益。

提出的方法

• 在 NEJM 和 JAMA 案例挑战上评估生物医学与通用型LLM（347道 NEJM 问题，140道 JAMA 问题）。
• 评估 CLUE 的 MeDiSumQA、MeDiSumCode、MedNLI、MeQSum、ProblemSummary 和 LongHealth 基准。
• 在统一提示和推理设置下，对各任务使用固定的评估指标（准确率、F1、ROUGE、BERTScore）。
• 涵盖不同尺寸与架构的模型（如 Llama、Mistral、OpenBioLLM 等）及其聊天/指令版本。
• 确保基准测试大概率来自生物医学微调数据之外，以避免数据泄露。

Figure 1: Comparative performance of the domain specific and general-purpose base models on the two clinical case vignette series. Bars represent the respective metric and error bars represent 95% confidence intervals obtained through bootstrapping.

实验结果

研究问题

• RQ1 生物医学微调的LLM在未见的临床病例数据上会超过通用型LLM吗？
• RQ2 在信息提取、编码与摘要等任务中，领域特定的LLM与通用模型相比的表现如何？
• RQ3 在长篇临床文档和易产生幻觉的任务中，通用模型的优势是否一致？
• RQ4 在医疗保健的LLM中，检索增强生成方法是否可能比领域特定微调更有效？

主要发现

• 在 JAMA 与 NEJM 案例挑战中，若干通用模型（例如 OpenBioLLM-70B、Llama-3-70B-Instruct）取得了最高准确率（例如 66-74%）。
• Llama-3-8B-Instruct 通常优于生物医学模型（例如 NEJM 64%/57% 对 18%，JAMA 64% 对 18%）。
• 在 MedNLI、ProblemSummary 与 MeQSum 中，所有生物医学 LLM 均不如其通用对手。
• MeDiSumCode 及部分长健康任务强调通用模型在深度知识任务与更长上下文处理方面的更高实力。
• LongHealth 任务结果显示生物医学模型产生幻觉更多，而通用模型在与幻觉相关的评估中表现相对更好。
• 总体而言，较大模型在生物医学与通用变体之间的性能差距更小，表明单纯微调可能不足以实现领域适应。