[Paper Review] Qibo: A Large Language Model for Traditional Chinese Medicine

Heyi Zhang, Xin Wang+6 more|arXiv (Cornell University)|Mar 24, 2024

Traditional Chinese Medicine Studies13 citations

TL;DR

Qibo is a Chinese-LLaMA-based LLM trained from pre-training to supervised fine-tuning on a custom Traditional Chinese Medicine corpus, plus a dedicated evaluation benchmark to assess TCM understanding and application.

ABSTRACT

Large Language Models (LLMs) has made significant progress in a number of professional fields, including medicine, law, and finance. However, in traditional Chinese medicine (TCM), there are challenges such as the essential differences between theory and modern medicine, the lack of specialized corpus resources, and the fact that relying only on supervised fine-tuning may lead to overconfident predictions. To address these challenges, we propose a two-stage training approach that combines continuous pre-training and supervised fine-tuning. A notable contribution of our study is the processing of a 2GB corpus dedicated to TCM, constructing pre-training and instruction fine-tuning datasets for TCM, respectively. In addition, we have developed Qibo-Benchmark, a tool that evaluates the performance of LLM in the TCM on multiple dimensions, including subjective, objective, and three TCM NLP tasks. The medical LLM trained with our pipeline, named $ extbf{Qibo}$, exhibits significant performance boosts. Compared to the baselines, the average subjective win rate is 63%, the average objective accuracy improved by 23% to 58%, and the Rouge-L scores for the three TCM NLP tasks are 0.72, 0.61, and 0.55. Finally, we propose a pipline to apply Qibo to TCM consultation and demonstrate the model performance through the case study.

Motivation & Objective

• Motivate the creation of a domain-specific LLM for Traditional Chinese Medicine (TCM) to address gaps between TCM theory and modern AI models.
• Develop a complete training pipeline (pre-training to SFT) for a TCM-focused LLM.
• Construct a high-quality, diverse TCM corpus and data processing rules for better domain knowledge capture.
• Create Qibo-benchmark to quantitatively evaluate LLMs in TCM knowledge, reasoning, and safety.

Proposed method

• Continuous pre-training on a diverse TCM and modern medical corpus to instill domain knowledge and diagnostic reasoning in Chinese-LLaMA.
• Supervised fine-tuning (SFT) using multi-source TCM dialogue datasets translated into an Alpaca-like conversation format.
• Incorporation of four data sources for SFT: TCM single-turn and multi-turn dialogues, TCM NLP task data, and general medical dialogues, ensuring domain relevance and generalization.
• A data processing pipeline with multi-granularity cleaning (character- and paragraph-level) and manual validation to create a high-quality training corpus.
• Evaluation of safety, professionalism, and fluency via GPT-4 and human expert assessments, plus objective NLP task benchmarks.

Experimental results

Research questions

• RQ1 How effectively can a LLaMA-based model be trained from pre-training to SFT to specialize in Traditional Chinese Medicine?
• RQ2 Does a domain-specific corpus and evaluation benchmark improve the model's TCM understanding, dialectics, and prescription recognition compared to general medical LLMs?
• RQ3 Can Qibo outperform existing Chinese medical LLMs in TCM-specific tasks and multi-turn dialogues while maintaining safety and professionalism?
• RQ4 What are the limitations and safety considerations when deploying a TCM-focused LLM in practice?

Key findings

• Qibo demonstrates strong domain performance in traditional Chinese medicine across subjective (professionalism, safety, fluency) and objective assessments.
• Qibo-13B yields higher accuracy than Qibo-7B on multi-turn TCM evaluation tasks, indicating scale benefits within the framework.
• In TCM NLP tasks, Qibo often surpasses baseline medical models but may lag task-specific models optimized for those tasks.
• The authors provide a dedicated Qibo-benchmark enabling quantitative comparison of TCM understanding and application across models.
• The model acknowledges limitations, including potential inaccuracies and the need for cautious real-world medical use, and points to future safety and multi-modal integration work.