[Paper Review] MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation

Seon‐Ok Kim|ArXiv.org|Feb 5, 2025

Biomedical Text Mining and Ontologies5 citations

TL;DR

MedBioLM combines domain-specific fine-tuning and retrieval-augmented generation (RAG) to improve biomedical QA across closed-ended, long-form, and short-form tasks, achieving higher accuracy than base models on key benchmarks and showing RAG enhances factuality in retrieval-dependent queries.

ABSTRACT

Large Language Models (LLMs) have demonstrated impressive capabilities across natural language processing tasks. However, their application to specialized domains such as medicine and biology requires further optimization to ensure factual accuracy, reliability, and contextual depth. We introduce MedBioLM, a domain-adapted biomedical question-answering model designed to enhance both short-form and long-form queries. By integrating fine-tuning and retrieval-augmented generation (RAG), MedBioLM dynamically incorporates domain-specific knowledge, improving reasoning abilities and factual accuracy. To evaluate its effectiveness, we fine-tuned the model on diverse biomedical QA datasets, covering structured multiple-choice assessments and complex clinical reasoning tasks. Fine-tuning significantly improves accuracy on benchmark datasets, while RAG enhances factual consistency. These results highlight the potential of domain-optimized LLMs in advancing biomedical research, medical education, and clinical decision support.

Motivation & Objective

• Motivate improved factual accuracy, reliability, and contextual depth in biomedical QA using domain-adapted LLMs.
• Evaluate the impact of fine-tuning, RAG, and prompt engineering across multiple QA formats (closed-ended, long-form, short-form).
• Assess performance on diverse biomedical QA datasets and determine conditions under which each optimization strategy helps most.

Proposed method

• Fine-tune a biomedical LLM on diverse QA datasets to improve domain-specific reasoning and factual accuracy.
• Integrate Retrieval-Augmented Generation (RAG) with a keyword-based, structured index for precise external knowledge retrieval.
• Apply prompt engineering to tailor system prompts and decoding parameters per QA format (closed-ended, long-form, short-form).
• Use Azure-based infrastructure for scalable fine-tuning and inference optimization.
• Evaluate using closed-ended accuracy and text-generation metrics (ROUGE, BLEU, BERTScore, BLEURT) across datasets.
• Compare against base models including GPT-4o, GPT-4, and GPT-3.5 to quantify gains from fine-tuning and RAG.

Figure 1: Comparative performance of MedBioLM and base models on closed-ended and short-form biomedical QA tasks, highlighting the benefits of fine-tuning.

Experimental results

Research questions

• RQ1 How does domain-specific fine-tuning affect accuracy on closed-ended biomedical QA datasets (MedQA, PubMedQA, BioASQ)?
• RQ2 What is the impact of Retrieval-Augmented Generation (RAG) on factual accuracy and lexical similarity in biomedical QA?
• RQ3 How do prompt engineering and decoding parameters influence short-form and long-form biomedical answer quality?
• RQ4 Do fine-tuned models outperform base models across multiple QA formats and datasets, and under what conditions does RAG add value?
• RQ5 Does GPT-4o benefit from domain adaptation compared to GPT-4 and GPT-3.5 in biomedical QA?

Key findings

• Fine-tuned MedBioLM achieves 88.0% accuracy on MedQA, 78.9% on PubMedQA, and 96.0% on BioASQ, outperforming GPT-4o and GPT-3.5 on MedQA and PubMedQA, with near-perfect BioASQ performance.
• RAG enhances short-form QA metrics, increasing ROUGE-1 and other lexical similarity measures, though fine-tuning has a stronger impact overall on short- and long-form outputs.
• Long-form QA shows substantial gains from fine-tuning in MedicationQA (ROUGE-1: 24.69; BLEU: 2.49; BERTScore: 8.98), while LiveQA results suggest potential overfitting in some cases.
• Short-form QA results indicate fine-tuned GPT-4o substantially outperforms the base model (ROUGE-1: 43.17 vs. 4.35; BLEU: 11.55 vs. 0.28), with RAG offering limited additional benefits when fine-tuning is applied.
• A comparative pairwise evaluation reveals GPT-4o often yields higher overall accuracy, while MedBioLM excels in coherence and succinctness in certain cases, highlighting complementary strengths.
• BLEURT scores remain largely negative for long-form generation across models, indicating ongoing challenges in producing human-like long-form responses.

Figure 2: Overview of our approach for optimizing large language models (LLMs) in biomedical question answering, integrating fine-tuning, retrieval-augmented generation (RAG), and prompt engineering to enhance performance across different QA formats.