[论文解读] HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge

Haochun Wang, Chi Liu+5 more|arXiv (Cornell University)|Apr 14, 2023

Topic Modeling被引用 94

一句话总结

HuaTuo 是一个基于 LLaMA-7B 的开源中国生物医学大语言模型，使用源自 CMeKG 的基于知识的指令数据进行微调，以提高中文医学问答中的医学知识可靠性。

ABSTRACT

Large Language Models (LLMs), such as the LLaMA model, have demonstrated their effectiveness in various general-domain natural language processing (NLP) tasks. Nevertheless, LLMs have not yet performed optimally in biomedical domain tasks due to the need for medical expertise in the responses. In response to this challenge, we propose HuaTuo, a LLaMA-based model that has been supervised-fine-tuned with generated QA (Question-Answer) instances. The experimental results demonstrate that HuaTuo generates responses that possess more reliable medical knowledge. Our proposed HuaTuo model is accessible at https://github.com/SCIR-HI/Huatuo-Llama-Med-Chinese.

研究动机与目标

• Address the gap of general-domain LLMs in biomedical accuracy for Chinese language contexts.
• Leverage structured and unstructured Chinese medical knowledge from CMeKG to guide model tuning.
• Create and use knowledge-based instruction data for supervised fine-tuning to improve factual correctness.
• Introduce a domain-specific evaluation metric (SUS) assessing safety, usability, and smoothness.

提出的方法

• Use LLaMA-7B as the base model.
• Incorporate structured and unstructured knowledge from the Chinese medical knowledge graph (CMeKG).
• Generate over 8,000 instruction data instances from knowledge graph knowledge for supervised fine-tuning.
• Perform instruction/data-driven supervised fine-tuning to align the model with medical-domain facts.
• Develop and employ the SUS metric to evaluate safety, usability, and smoothness in biomedical responses.
• Compare HuaTuo against baselines LLaMA, Alpaca, and ChatGLM on Chinese biomedical dialogue tasks.

实验结果

研究问题

• RQ1 Does incorporating CMeKG-based knowledge improve medical knowledge usability in Chinese biomedical QA?
• RQ2 How does HuaTuo compare to LLaMA, Alpaca, and ChatGLM in safety, usability, and smoothness on Chinese medical dialogue tasks?
• RQ3 What is the impact of knowledge-based instruction data on biomedical factual accuracy and reliability?
• RQ4 Can an open-source Chinese biomedical LLM be effectively tuned with domain knowledge without compromising safety?

主要发现

• HuaTuo improves usability of medical-domain responses compared to baselines while maintaining safety.
• On SUS evaluation, HuaTuo scores 2.88 (Safety), 2.12 (Usability), and 2.47 (Smoothness) closely aligning with or surpassing Alpaca and ChatGLM in usability and smoothness.
• LLaMA achieves the highest safety score but provides less informative responses, leading to lower usability.
• HuaTuo demonstrates more reliable medical knowledge through knowledge-based instruction fine-tuning.
• The model is released as open-source at the provided GitHub repository.
• The SUS-based evaluation emphasizes domain-relevant assessment beyond traditional BLEU/Rouge metrics.