[Paper Review] Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approaches

Christophe Clément, Praveen K Kanithi+14 more|arXiv (Cornell University)|Apr 23, 2024

Biomedical and Engineering Education12 citations

TL;DR

This paper compares full-parameter fine-tuning and parameter-efficient fine-tuning (notably LoRA) for medical LLMs built on Llama-2, reporting Med42 achieving 72% accuracy on USMLE and detailing decontamination and openness of models.

ABSTRACT

This study presents a comprehensive analysis and comparison of two predominant fine-tuning methodologies - full-parameter fine-tuning and parameter-efficient tuning - within the context of medical Large Language Models (LLMs). We developed and refined a series of LLMs, based on the Llama-2 architecture, specifically designed to enhance medical knowledge retrieval, reasoning, and question-answering capabilities. Our experiments systematically evaluate the effectiveness of these tuning strategies across various well-known medical benchmarks. Notably, our medical LLM Med42 showed an accuracy level of 72% on the US Medical Licensing Examination (USMLE) datasets, setting a new standard in performance for openly available medical LLMs. Through this comparative analysis, we aim to identify the most effective and efficient method for fine-tuning LLMs in the medical domain, thereby contributing significantly to the advancement of AI-driven healthcare applications.

Motivation & Objective

• Motivate the need for domain-specific fine-tuning of medical LLMs to improve retrieval, reasoning, and QA.
• Develop and evaluate medical LLMs (Med42) based on Llama-2 with full-parameter and LoRA-style tuning.
• Assess performance across established medical benchmarks and analyze training efficiency and contamination risks.
• Provide open-access deployment of the best-performing model to advance AI in healthcare.

Proposed method

• Construct medical instruction-tuned datasets from Stack Exchange medical forums and selective general-domain medical content.
• Fine-tune 7B and 70B Llama-2 models with full-parameter tuning and with LoRA-based parameter-efficient tuning.
• Apply mask loss by concatenating system/prompt/assistant samples and autoregressively train only the response tokens.
• Use AdamW with cosine LR schedule, warmup, and standard regularization; specify epoch counts and learning rates for FP-FT and LoRA.
• Evaluate zero-shot performance on diverse medical benchmarks using Eleuther AI’s evaluation harness, with contaminant sample decontamination.
• Release Med42 (70B) on HuggingFace for reproducibility and open research.

Figure 1: Performance of 7-billion (left) and 70-billion (right) parameter models on various medical-related benchmark datasets (in zero-shot setting). Performance results (accuracy) are displayed in % for the base and fine-tuned models.

Experimental results

Research questions

• RQ1 How do full-parameter fine-tuning and LoRA-based parameter-efficient fine-tuning compare in medical QA and reasoning tasks?
• RQ2 Does model size (7B vs 70B) influence the relative gains from full-parameter vs. LoRA fine-tuning in medical domains?
• RQ3 What is the impact of dataset decontamination on reported benchmark performance?
• RQ4 How does Med42 compare to other open medical LLMs on standard benchmarks and USMLE-style questions?

Key findings

• Fine-tuned models outperform their base counterparts across medical benchmarks.
• Full-parameter fine-tuning generally outperforms LoRA in most datasets.
• LoRA achieves performance close to full-parameter fine-tuning, offering a viable resource-efficient alternative.
• Decontamination caused only small changes in accuracy for the larger fine-tuned models, indicating robustness of results.
• Med42-70B achieves strong zero-shot performance on USMLE-related tasks and outperforms several open medical LLMs in comparisons.
• The study emphasizes open licensing and reproducibility, releasing Med42 and the evaluation framework.

Figure 2: Accuracy change after decontamination for both (70b) fine-tuned models (shown in %).