[Paper Review] MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data

Tianyu Han, Lisa C. Adams+7 more|arXiv (Cornell University)|Apr 14, 2023

Artificial Intelligence in Healthcare and Education104 citations

TL;DR

The paper presents an open-source medical LLM fine-tuning dataset and evaluates fine-tuned models versus pre-trained baselines on USMLE self-assessment tasks, showing that larger pre-trained models with high-quality medical data improve performance. It also discusses training techniques like LoRA and 8-bit methods and highlights privacy advantages of on-site deployment.

ABSTRACT

As large language models (LLMs) like OpenAI's GPT series continue to make strides, we witness the emergence of artificial intelligence applications in an ever-expanding range of fields. In medicine, these LLMs hold considerable promise for improving medical workflows, diagnostics, patient care, and education. Yet, there is an urgent need for open-source models that can be deployed on-premises to safeguard patient privacy. In our work, we present an innovative dataset consisting of over 160,000 entries, specifically crafted to fine-tune LLMs for effective medical applications. We investigate the impact of fine-tuning these datasets on publicly accessible pre-trained LLMs, and subsequently, we juxtapose the performance of pre-trained-only models against the fine-tuned models concerning the examinations that future medical doctors must pass to achieve certification.

Motivation & Objective

• Create an open-access, on-premises-friendly collection of medical language models and training data.
• Evaluate the impact of fine-tuning on publicly available LLMs for medical tasks.
• Assess model performance on USMLE Step 1–3 self-assessments to gauge medical competence.
• Provide guidance on privacy, bias, and reliability considerations for medical AI deployment.

Proposed method

• Assemble Medical Meadow, a diverse mix of medical NLP tasks and internet-sourced data reformatted for instruction-tuning.
• Fine-tune LLaMA base models (7B and 13B) using full fine-tuning and parameter-efficient methods (LoRA, 8-bit training).
• Train with different learning rates and epochs, using a cosine scheduler and gradient accumulation for effective batch size 256.
• Evaluate zero-shot performance on USMLE Step 1, 2, and 3 self-assessment datasets, excluding image-based questions, enforcing an 'Option: Answer' output format.
• Compare full fine-tuning against LoRA and 8-bit variants to assess trade-offs between performance and compute.

Experimental results

Research questions

• RQ1 Do fine-tuned medical LLMs outperform their pre-trained-only counterparts on USMLE self-assessment tasks?
• RQ2 How does model size (7B vs. 13B) affect medical knowledge performance after fine-tuning?
• RQ3 What is the impact of parameter-efficient tuning (LoRA, 8-bit) on accuracy relative to full fine-tuning?
• RQ4 Does increasing data quality and domain specificity in fine-tuning data improve USMLE-style performance?

Key findings

• Fine-tuned MedAlpaca models outperform pre-trained-only baselines across USMLE Step 1, 2, and 3.
• MedAlpaca 13b achieves higher raw scores (Step 1: 0.473, Step 2: 0.477, Step 3: 0.602) than smaller models.
• LoRA and 8-bit fine-tuning speed training but generally reduce accuracy compared with vanilla fine-tuning.
• Performance improves with larger pre-trained models and high-quality medical data.
• Open-source models enable on-site deployment to address privacy concerns in medical contexts.