[Paper Review] PMC-LLaMA: Towards Building Open-source Language Models for Medicine

Chaoyi Wu, Lin, Weixiong+4 more|arXiv (Cornell University)|Apr 27, 2023

Topic Modeling56 citations

TL;DR

PMC-LLaMA is an open-source, 13B medical-focused language model built from LLaMA via data-centric knowledge injection and medical instruction tuning, surpassing ChatGPT on several medical QA benchmarks while remaining lightweight.

ABSTRACT

Recently, Large Language Models (LLMs) have showcased remarkable capabilities in natural language understanding. While demonstrating proficiency in everyday conversations and question-answering situations, these models frequently struggle in domains that require precision, such as medical applications, due to their lack of domain-specific knowledge. In this paper, we describe the procedure for building a powerful, open-source language model specifically designed for medicine applications, termed as PMC-LLaMA. Our contributions are threefold: (i) we systematically investigate the process of adapting a general-purpose foundation language model towards medical domain, this involves data-centric knowledge injection through the integration of 4.8M biomedical academic papers and 30K medical textbooks, as well as comprehensive fine-tuning for alignment with domain-specific instructions; (ii) we contribute a large-scale, comprehensive dataset for instruction tuning. This dataset encompasses medical question-answering (QA), rationale for reasoning, and conversational dialogues, comprising a total of 202M tokens; (iii) we conduct thorough ablation studies to demonstrate the effectiveness of each proposed component. While evaluating on various public medical question-answering benchmarks, our lightweight PMCLLaMA, which consists of only 13 billion parameters, exhibits superior performance, even surpassing ChatGPT. All models, codes, datasets can be found in https://github.com/chaoyi-wu/PMC-LLaMA.

Motivation & Objective

• Investigate adapting a general-purpose LLM to the medical domain with data-centric knowledge injection.
• Assemble a large medical corpus (MedC-K) and a medical instruction-tuning dataset (MedC-I) for domain alignment.
• Evaluate PMC-LLaMA on standard medical QA benchmarks and analyze ablations to identify key contributing components.

Proposed method

• Inject medical knowledge by training on 4.8M biomedical papers and 30K textbooks to form MedC-K.
• Perform medical-specific instruction tuning on MedC-I (202M tokens) to align with clinical use cases.
• Use a two-stage training: knowledge injection (auto-regressive loss) followed by instruction tuning (I, R format, rationale, and knowledge-graph prompts).
• Incorporate three data sources for instruction tuning: medical conversation data, rationale QA, and knowledge-graph prompting (UMLS).
• Evaluate using three public medical QA benchmarks (PubMedQA, MedMCQA, USMLE) and conduct ablations across model size, data injection, and instruction tuning.

Experimental results

Research questions

• RQ1 Can a 13B open-source LLM be competitive with or surpass larger closed models (e.g., ChatGPT) in medical QA after data-centric knowledge injection and medical-specific instruction tuning?
• RQ2 How do different components (papers/books knowledge injection, rationale QA, conversation data, and knowledge-graph prompting) contribute to medical QA performance?
• RQ3 What is the impact of model scale and training regimen on medical QA benchmarks?

Key findings

• PMC-LLaMA-13B with full medical knowledge injection and instruction tuning achieves higher average QA accuracy (64.43) across MedQA, MedMCQA, and PubMedQA compared to several baselines.
• In ablations, adding books improves MedQA, MedMCQA, and PubMedQA by up to about 1.0–2.9 points; adding papers also yields gains; larger model size (13B) improves performance over 7B.
• Incorporating medical rationale QA and knowledge-graph prompting during instruction tuning yields performance boosts (e.g., MedQA: 49.32% to 54.43%, and knowledge graph adds ~1.93%); conversations plus rationale data significantly enhance zero-shot QA.
• PMC-LLaMA-13B with full setup attains higher average QA accuracy than ChatGPT on the evaluated benchmarks (average 64.43 vs. 54.97 for ChatGPT on MedQA/MedMCQA/PubMedQA).
• The model remains open-source with released code and data at the authors’ GitHub.