[Paper Review] ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge

Yunxiang Li, Li Zihan+4 more|arXiv (Cornell University)|Mar 24, 2023

Machine Learning in Healthcare33 citations

TL;DR

ChatDoctor fine-tunes LLaMA on 100k real patient-physician dialogues and equips it with autonomous external knowledge retrieval from online/offline sources to improve medical response accuracy compared with ChatGPT.

ABSTRACT

The primary aim of this research was to address the limitations observed in the medical knowledge of prevalent large language models (LLMs) such as ChatGPT, by creating a specialized language model with enhanced accuracy in medical advice. We achieved this by adapting and refining the large language model meta-AI (LLaMA) using a large dataset of 100,000 patient-doctor dialogues sourced from a widely used online medical consultation platform. These conversations were cleaned and anonymized to respect privacy concerns. In addition to the model refinement, we incorporated a self-directed information retrieval mechanism, allowing the model to access and utilize real-time information from online sources like Wikipedia and data from curated offline medical databases. The fine-tuning of the model with real-world patient-doctor interactions significantly improved the model's ability to understand patient needs and provide informed advice. By equipping the model with self-directed information retrieval from reliable online and offline sources, we observed substantial improvements in the accuracy of its responses. Our proposed ChatDoctor, represents a significant advancement in medical LLMs, demonstrating a significant improvement in understanding patient inquiries and providing accurate advice. Given the high stakes and low error tolerance in the medical field, such enhancements in providing accurate and reliable information are not only beneficial but essential.

Motivation & Objective

• Improve medical dialogue accuracy by fine-tuning a language model on real-world patient-physician conversations.
• Enhance reliability by integrating an external knowledge brain (online/offline) for real-time information retrieval.
• Demonstrate improvements over a general-domain model (ChatGPT) in precision, recall, and F1 on medical queries.

Proposed method

• Fine-tune LLaMA-7B on HealthCareMagic-100k patient-doctor dialogues following Alpaca-style instruction tuning.
• Create an external knowledge brain from MedlinePlus-derived disease data and supplementary sources like Wikipedia.
• Develop autonomous keyword mining prompts to extract key terms from queries for knowledge retrieval.
• Implement a top-ranked, keyword-driven retrieval system with sectioned text and token-limited processing for the retrieved content.
• Prompt the model to read retrieved knowledge sections and generate a final, informed response.
• Evaluate using iCliniq-derived questions with human physician responses as ground truth, applying BERTScore for Precision, Recall, and F1.

Experimental results

Research questions

• RQ1 Can a medical-domain-tuned LLM outperform a general-domain model (ChatGPT) on medical dialogue tasks?
• RQ2 Does adding an autonomous external knowledge retrieval mechanism improve answer accuracy and up-to-dateness for medical questions?
• RQ3 How does ChatDoctor perform on queries involving relatively new terms or diseases not in the training set (e.g., Mpox, Daybue)?

Key findings

• ChatDoctor outperforms ChatGPT on BERTScore Precision, Recall, and F1 in the reported evaluation.
• The autonomous knowledge retrieval enables correct responses to newer terms (e.g., Mpox) and newly approved medications (e.g., Daybue).
• Qualitative examples show ChatDoctor provides more specialized and corroborated medical guidance than ChatGPT in several scenarios.
• The model uses HealthCareMagic-100k for fine-tuning and iCliniq for testing data, with improvements demonstrated in the presented metrics.