[Paper Review] Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding

Augustin Toma, Patrick R. Lawler+4 more|arXiv (Cornell University)|May 19, 2023

Artificial Intelligence in Healthcare and Education35 citations

TL;DR

Clinical Camel is an open, expert-level medical LLM fine-tuned from LLaMA-2 using QLoRA, achieving state-of-the-art performance among open medical LLMs and surpassing GPT-3.5 in several benchmarks, with novel dialogue-based knowledge encoding to convert dense medical texts into conversations.

ABSTRACT

We present Clinical Camel, an open large language model (LLM) explicitly tailored for clinical research. Fine-tuned from LLaMA-2 using QLoRA, Clinical Camel achieves state-of-the-art performance across medical benchmarks among openly available medical LLMs. Leveraging efficient single-GPU training, Clinical Camel surpasses GPT-3.5 in five-shot evaluations on all assessed benchmarks, including 64.3% on the USMLE Sample Exam (compared to 58.5% for GPT-3.5), 77.9% on PubMedQA (compared to 60.2%), 60.7% on MedQA (compared to 53.6%), and 54.2% on MedMCQA (compared to 51.0%). In addition to these benchmarks, Clinical Camel demonstrates its broader capabilities, such as synthesizing plausible clinical notes. This work introduces dialogue-based knowledge encoding, a novel method to synthesize conversational data from dense medical texts. While benchmark results are encouraging, extensive and rigorous human evaluation across diverse clinical scenarios is imperative to ascertain safety before implementation. By openly sharing Clinical Camel, we hope to foster transparent and collaborative research, working towards the safe integration of LLMs within the healthcare domain. Significant challenges concerning reliability, bias, and the potential for outdated knowledge persist. Nonetheless, the transparency provided by an open approach reinforces the scientific rigor essential for future clinical applications.

Motivation & Objective

• Motivate the need for open, high-performance medical LLMs for research and safe clinical deployment.
• Present Clinical Camel as an open alternative that rivals proprietary models on standard medical benchmarks.
• Introduce DBKE (Dialogue-Based Knowledge Encoding) to convert dense medical literature into conversational data for training.
• Demonstrate efficient fine-tuning on 13B and 70B parameter variants using QLoRA on a single GPU.

Proposed method

• Fine-tune LLaMA-2 (13B and 70B) with QLoRA on a single commercial GPU using masking of human inputs.
• Develop DBKE to transform dense clinical texts into multi-turn dialogues via a teacher-student fine-tuning setup.
• Create training data from ShareGPT, MedQA, and PubMed/Open-access clinical articles pre-2021 transformed into dialogues.
• Train for one epoch with 4096 token sequences and 64 LoRA rank across modules.
• Evaluate using EleutherAI evaluation framework to report accuracy on zero- and five-shot medical benchmarks.

Experimental results

Research questions

• RQ1 Can an open medical LLM match or exceed GPT-3.5 on standard medical benchmarks with efficient fine-tuning?
• RQ2 Does the Dialogue-Based Knowledge Encoding method improve recall and conversational performance for domain-specific medical texts?
• RQ3 What is the performance gap between open medical LLMs and top proprietary models (GPT-4, Med-PaLM 2) on medical QA benchmarks?
• RQ4 Is it feasible to train a high-performing medical LLM on a single GPU while maintaining competitive context length and capabilities?

Key findings

• In five-shot evaluations, Clinical Camel surpasses GPT-3.5 on all assessed benchmarks: USMLE Sample Exam 64.3% vs 58.5%, PubMedQA 77.9% vs 60.2%, MedQA 60.7% vs 53.6%, MedMCQA 54.2% vs 51.0%.
• Clinical Camel outperforms GPT-3.5 across multiple MMLU medical domains in zero-shot and five-shot settings, while still lagging behind GPT-4 and Med-PaLM 2 overall.
• On PubMedQA, Clinical Camel surpasses GPT-4 in five-shot evaluations (GPT-4 score cited as 74.4 vs 79.2 for Med-PaLM 2 and 77.9 for Clinical Camel).
• The model supports longer context (4096 tokens) enabling generation of plausible clinical notes from long dialogues.
• DBKE enables converting dense medical literature into dialogues to improve model recall and alignment while reducing data and compute requirements.