[Paper Review] HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge

Haochun Wang, Chi Liu+5 more|arXiv (Cornell University)|Apr 14, 2023

Topic Modeling94 citations

TL;DR

HuaTuo is an open-source LLaMA-7B-based Chinese biomedical LLM tuned with knowledge-based instruction data derived from CMeKG to improve medical knowledge reliability in Chinese medical QA.

ABSTRACT

Large Language Models (LLMs), such as the LLaMA model, have demonstrated their effectiveness in various general-domain natural language processing (NLP) tasks. Nevertheless, LLMs have not yet performed optimally in biomedical domain tasks due to the need for medical expertise in the responses. In response to this challenge, we propose HuaTuo, a LLaMA-based model that has been supervised-fine-tuned with generated QA (Question-Answer) instances. The experimental results demonstrate that HuaTuo generates responses that possess more reliable medical knowledge. Our proposed HuaTuo model is accessible at https://github.com/SCIR-HI/Huatuo-Llama-Med-Chinese.

Motivation & Objective

• Address the gap of general-domain LLMs in biomedical accuracy for Chinese language contexts.
• Leverage structured and unstructured Chinese medical knowledge from CMeKG to guide model tuning.
• Create and use knowledge-based instruction data for supervised fine-tuning to improve factual correctness.
• Introduce a domain-specific evaluation metric (SUS) assessing safety, usability, and smoothness.

Proposed method

• Use LLaMA-7B as the base model.
• Incorporate structured and unstructured knowledge from the Chinese medical knowledge graph (CMeKG).
• Generate over 8,000 instruction data instances from knowledge graph knowledge for supervised fine-tuning.
• Perform instruction/data-driven supervised fine-tuning to align the model with medical-domain facts.
• Develop and employ the SUS metric to evaluate safety, usability, and smoothness in biomedical responses.
• Compare HuaTuo against baselines LLaMA, Alpaca, and ChatGLM on Chinese biomedical dialogue tasks.

Experimental results

Research questions

• RQ1 Does incorporating CMeKG-based knowledge improve medical knowledge usability in Chinese biomedical QA?
• RQ2 How does HuaTuo compare to LLaMA, Alpaca, and ChatGLM in safety, usability, and smoothness on Chinese medical dialogue tasks?
• RQ3 What is the impact of knowledge-based instruction data on biomedical factual accuracy and reliability?
• RQ4 Can an open-source Chinese biomedical LLM be effectively tuned with domain knowledge without compromising safety?

Key findings

• HuaTuo improves usability of medical-domain responses compared to baselines while maintaining safety.
• On SUS evaluation, HuaTuo scores 2.88 (Safety), 2.12 (Usability), and 2.47 (Smoothness) closely aligning with or surpassing Alpaca and ChatGLM in usability and smoothness.
• LLaMA achieves the highest safety score but provides less informative responses, leading to lower usability.
• HuaTuo demonstrates more reliable medical knowledge through knowledge-based instruction fine-tuning.
• The model is released as open-source at the provided GitHub repository.
• The SUS-based evaluation emphasizes domain-relevant assessment beyond traditional BLEU/Rouge metrics.