[Paper Review] BianCang: A Traditional Chinese Medicine Large Language Model

Sibo Wei, Xueping Peng+9 more|arXiv (Cornell University)|Nov 17, 2024

Traditional Chinese Medicine Studies5 citations

TL;DR

BianCang is a two-stage trained TCM-specific LLM built from Qwen that injects extensive traditional Chinese medicine knowledge via continuous pre-training and aligns it with real-world tasks through supervised fine-tuning, achieving strong syndrome differentiation and diagnosis performance across 11 test sets.

ABSTRACT

The surge of large language models (LLMs) has driven significant progress in medical applications, including traditional Chinese medicine (TCM). However, current medical LLMs struggle with TCM diagnosis and syndrome differentiation due to substantial differences between TCM and modern medical theory, and the scarcity of specialized, high-quality corpora. To this end, in this paper we propose BianCang, a TCM-specific LLM, using a two-stage training process that first injects domain-specific knowledge and then aligns it through targeted stimulation to enhance diagnostic and differentiation capabilities. Specifically, we constructed pre-training corpora, instruction-aligned datasets based on real hospital records, and the ChP-TCM dataset derived from the Pharmacopoeia of the People's Republic of China. We compiled extensive TCM and medical corpora for continual pre-training and supervised fine-tuning, building a comprehensive dataset to refine the model's understanding of TCM. Evaluations across 11 test sets involving 31 models and 4 tasks demonstrate the effectiveness of BianCang, offering valuable insights for future research. Code, datasets, and models are available on https://github.com/QLU-NLP/BianCang.

Motivation & Objective

• Motivate the development of a TCM-focused LLM able to perform accurate syndrome differentiation and disease diagnosis.
• Inject extensive TCM and medical knowledge through continuous pre-training to establish a solid knowledge foundation.
• Align and activate embedded knowledge via targeted supervised fine-tuning on diverse TCM instructions and real-world data.
• Evaluate BianCang against multiple baselines across syndrome differentiation, diagnosis, and exams to demonstrate robustness.

Proposed method

• Two-stage training on Qwen-based foundations: continuous pre-training with large TCM/medical corpora and real records, followed by supervised fine-tuning with domain-specific instructions.
• Construction of a comprehensive pre-training dataset including MedicalBooks, Encyclopedia, Literature, Pharmacopoeia, MedicalRecords, TCMSD&DD Records, and specialized knowledge bases.
• Development of ChP-TCM-based instruction datasets (ChP-KnowledgeQA, ChP-PrescriptionWriting) plus medical-record and multi-turn dialogue SFT datasets (DISC-Med-SFT, MLEC-SFT, TCM-SFT variants).
• Fine-tuning at 7B and 14B scales using full-parameter fine-tuning with SWIFT, applying regularization and training stability heuristics across two epochs on A100 GPUs.
• Evaluation across 11 test sets and 4 tasks, comparing with a broad set of baselines (medical and TCM LLMs, including GPT-4) to verify improvements in TCM syndrome differentiation, disease diagnosis, and exams.

Experimental results

Research questions

• RQ1 Can a TCM-focused LLM achieve superior syndrome differentiation and diagnosis using a two-stage training paradigm?
• RQ2 How does continuous pre-training with extensive TCM/medical corpora plus real patient data affect downstream SFT performance in TCM tasks?
• RQ3 Does targeted alignment through ChP-TCM instruction sets improve real-world diagnostic and treatment planning capabilities in TCM?
• RQ4 How does BianCang compare with existing open-source medical and TCM LLMs across multiple objective and subjective evaluation dimensions?

Key findings

• BianCang outperforms existing open-source TCM and Chinese medical LLMs on 11 test sets across 4 tasks (syndrome differentiation, disease diagnosis, exams, and medical knowledge).
• In direct inference mode on TCMSD, BianCang-Qwen2.5-7B-Instruct achieved 78.90% accuracy and 82.10% in CoT, substantially higher than the baseline Qwen2.5-7B.
• BianCang demonstrates robust performance gains in both zero-shot and few-shot exam settings, and shows strong subjective evaluations in professionalism, fluency, and safety per BC-Analytical assessment.
• The two-stage training approach (continuous pre-training plus SFT) improves consistency between embedded knowledge and instruction-aligned capabilities, enhancing performance on syndrome differentiation and diagnosis tasks.