[Paper Review] Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data

Felix J. Dorfner, Amin Dada+9 more|arXiv (Cornell University)|Aug 25, 2024

Cancer Genomics and Diagnostics5 citations

TL;DR

Biomedical LLMs fine-tuned on domain data generally underperform compared to general-purpose models on unseen medical data across multiple clinical tasks, suggesting limited value of biomedical fine-tuning and highlighting retrieval-augmented approaches as promising alternatives.

ABSTRACT

Large language models (LLMs) have shown potential in biomedical applications, leading to efforts to fine-tune them on domain-specific data. However, the effectiveness of this approach remains unclear. This study evaluates the performance of biomedically fine-tuned LLMs against their general-purpose counterparts on a variety of clinical tasks. We evaluated their performance on clinical case challenges from the New England Journal of Medicine (NEJM) and the Journal of the American Medical Association (JAMA) and on several clinical tasks (e.g., information extraction, document summarization, and clinical coding). Using benchmarks specifically chosen to be likely outside the fine-tuning datasets of biomedical models, we found that biomedical LLMs mostly perform inferior to their general-purpose counterparts, especially on tasks not focused on medical knowledge. While larger models showed similar performance on case tasks (e.g., OpenBioLLM-70B: 66.4% vs. Llama-3-70B-Instruct: 65% on JAMA cases), smaller biomedical models showed more pronounced underperformance (e.g., OpenBioLLM-8B: 30% vs. Llama-3-8B-Instruct: 64.3% on NEJM cases). Similar trends were observed across the CLUE (Clinical Language Understanding Evaluation) benchmark tasks, with general-purpose models often performing better on text generation, question answering, and coding tasks. Our results suggest that fine-tuning LLMs to biomedical data may not provide the expected benefits and may potentially lead to reduced performance, challenging prevailing assumptions about domain-specific adaptation of LLMs and highlighting the need for more rigorous evaluation frameworks in healthcare AI. Alternative approaches, such as retrieval-augmented generation, may be more effective in enhancing the biomedical capabilities of LLMs without compromising their general knowledge.

Motivation & Objective

• Assess whether biomedical fine-tuning improves LLM performance on unseen clinical data and tasks.
• Compare biomedically fine-tuned LLMs against general-purpose baselines across diverse clinical benchmarks.
• Investigate task areas where domain adaptation may or may not provide benefits in healthcare AI.

Proposed method

• Evaluate biomedical and general-purpose LLMs on NEJM and JAMA case challenges (347 NEJM, 140 JAMA questions).
• Assess MeDiSumQA, MeDiSumCode, MedNLI, MeQSum, ProblemSummary, and LongHealth benchmarks from CLUE.
• Use fixed evaluation metrics (accuracy, F1, ROUGE, BERTScore) across tasks with standardized prompting and inference settings.
• Include models across different sizes and architectures (Llama, Mistral, OpenBioLLM, etc.) and their chat/instruct variants.
• Ensure benchmarks are likely outside the biomedical fine-tuning data to avoid data leakage.

Figure 1: Comparative performance of the domain specific and general-purpose base models on the two clinical case vignette series. Bars represent the respective metric and error bars represent 95% confidence intervals obtained through bootstrapping.

Experimental results

Research questions

• RQ1 Do biomedical fine-tuned LLMs outperform general-purpose LLMs on unseen clinical case data?
• RQ2 How do domain-specific LLMs perform on information extraction, coding, and summarization tasks compared to generalist models?
• RQ3 Is the advantage of generalist models consistent across long-form clinical documents and hallucination-prone tasks?
• RQ4 Are retrieval-augmented generation approaches potentially more effective than domain-specific fine-tuning in healthcare LLMs?

Key findings

• On JAMA and NEJM case challenges, several generalist models (e.g., OpenBioLLM-70B, Llama-3-70B-Instruct) achieved top accuracies (e.g., 66-74%).
• Llama-3-8B-Instruct often outperformed biomedical models (e.g., 64-57% vs 18% on NEJM and 64% vs 18% on JAMA).
• Across MedNLI, ProblemSummary, and MeQSum, all biomedical LLMs underperformed compared to their generalist counterparts.
• MeDiSumCode and some long-health tasks emphasized higher strength of generalist models in depth knowledge tasks and longer context handling.
• LongHealth task results showed biomedical models exhibited more hallucinations and generalist models performed comparatively better on hallucination-related evaluation.
• Overall, larger models displayed smaller performance gaps between biomedical and generalist variants, suggesting fine-tuning alone may not be sufficient for domain adaptation.