[Paper Review] MedChatZH: a Better Medical Adviser Learns from Better Instructions

Yang Tan, Mingchen Li+3 more|arXiv (Cornell University)|Sep 3, 2023

Topic Modeling4 citations

TL;DR

MedChatZH is a Chinese medical dialogue model fine-tuned on a curated dataset of traditional Chinese medicine (TCM) texts and medical instruction pairs, achieving state-of-the-art performance in single-turn Chinese medical QA. It outperforms baselines using both automatic metrics (BLEU, ROUGE-L, GLEU) and a fine-tuned reward model, with code, model, and dataset released open-source for community use.

ABSTRACT

Generative large language models (LLMs) have shown great success in various applications, including question-answering (QA) and dialogue systems. However, in specialized domains like traditional Chinese medical QA, these models may perform unsatisfactorily without fine-tuning on domain-specific datasets. To address this, we introduce MedChatZH, a dialogue model designed specifically for traditional Chinese medical QA. Our model is pre-trained on Chinese traditional medical books and fine-tuned with a carefully curated medical instruction dataset. It outperforms several solid baselines on a real-world medical dialogue dataset. We release our model, code, and dataset on https://github.com/tyang816/MedChatZH to facilitate further research in the domain of traditional Chinese medicine and LLMs.

Motivation & Objective

• To address the poor performance of general-purpose LLMs in specialized domains like traditional Chinese medicine (TCM) by creating a domain-specific dialogue model.
• To improve Chinese medical question-answering by pre-training on a large corpus of traditional Chinese medical books and fine-tuning with high-quality instruction data.
• To mitigate hallucination and low-quality outputs in medical LLMs through a rigorous data curation pipeline that removes sensitive, irrelevant, and colloquial content.
• To release a reproducible, open-source model and dataset to support further research in Chinese medical LLMs.

Proposed method

• Pre-trained a Chinese-specific LLM on a large-scale collection of traditional Chinese medical texts to enhance domain-specific knowledge.
• Curated a high-quality medical instruction dataset using a multi-stage pipeline involving heuristic filtering, LLM-based data filtering, and reward model scoring to remove irrelevant, sensitive, or low-quality content.
• Fine-tuned the pre-trained model on the curated instruction dataset using instruction tuning to improve medical QA and dialogue generation capabilities.
• Evaluated model performance using standard NLP metrics (BLEU, ROUGE-L, GLEU) and a domain-specific reward model (Ziya-LLaMA-7B-Reward) for quality assessment.
• Conducted ablation studies to verify the impact of instruction fine-tuning on model performance.
• Released the trained model, code, and dataset on GitHub to ensure reproducibility and community access.

Experimental results

Research questions

• RQ1 Can a large language model fine-tuned on domain-specific traditional Chinese medical texts and instruction data achieve superior performance in Chinese medical dialogue compared to general-purpose or English-focused models?
• RQ2 How effective is a multi-stage data curation pipeline—including heuristic rules, LLM filtering, and reward model scoring—in improving the quality and safety of medical instruction data?
• RQ3 To what extent does instruction fine-tuning enhance the model’s ability to generate accurate, fluent, and medically appropriate responses in single-turn medical QA?
• RQ4 How does MedChatZH compare to strong baselines like HuatuoGPT and GPT-3.5-turbo in both automatic metrics and human-like reward scoring?

Key findings

• MedChatZH outperforms multiple strong baselines on a real-world Chinese medical dialogue benchmark, demonstrating state-of-the-art performance across multiple evaluation metrics.
• The ablation study confirms that fine-tuning with distilled, high-quality medical instruction data significantly improves the model’s medical QA capability.
• The use of a reward model (Ziya-LLaMA-7B-Reward) for evaluation provides a more nuanced and reliable assessment of response quality than traditional metrics alone.
• The model achieves strong performance on standard NLP metrics such as BLEU, ROUGE-L, and GLEU, indicating high fluency and n-gram overlap with reference answers.
• The model’s performance is robust across multiple evaluation criteria, including factual consistency, instruction adherence, and response coherence.
• The open-sourcing of the model, code, and dataset enables reproducibility and supports future research in Chinese medical LLMs.