[Paper Review] TCM-GPT: Efficient Pre-training of Large Language Models for Domain Adaptation in Traditional Chinese Medicine

Guoxing Yang, Jian‐Yu Shi+3 more|arXiv (Cornell University)|Nov 3, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes TCMDA, a domain-adaptive pre-training method for large language models in Traditional Chinese Medicine (TCM), using a 1-billion-token TCM-specific corpus (TCM-Corpus-1B) and LoRA fine-tuning. TCM-GPT-7B, the resulting 7B-parameter model, achieves 17% and 12% relative accuracy gains on TCM examination and diagnosis tasks, demonstrating the effectiveness of domain-specific pre-training in TCM.

ABSTRACT

Pre-training and fine-tuning have emerged as a promising paradigm across various natural language processing (NLP) tasks. The effectiveness of pretrained large language models (LLM) has witnessed further enhancement, holding potential for applications in the field of medicine, particularly in the context of Traditional Chinese Medicine (TCM). However, the application of these general models to specific domains often yields suboptimal results, primarily due to challenges like lack of domain knowledge, unique objectives, and computational efficiency. Furthermore, their effectiveness in specialized domains, such as Traditional Chinese Medicine, requires comprehensive evaluation. To address the above issues, we propose a novel domain specific TCMDA (TCM Domain Adaptation) approach, efficient pre-training with domain-specific corpus. Specifically, we first construct a large TCM-specific corpus, TCM-Corpus-1B, by identifying domain keywords and retreving from general corpus. Then, our TCMDA leverages the LoRA which freezes the pretrained model's weights and uses rank decomposition matrices to efficiently train specific dense layers for pre-training and fine-tuning, efficiently aligning the model with TCM-related tasks, namely TCM-GPT-7B. We further conducted extensive experiments on two TCM tasks, including TCM examination and TCM diagnosis. TCM-GPT-7B archived the best performance across both datasets, outperforming other models by relative increments of 17% and 12% in accuracy, respectively. To the best of our knowledge, our study represents the pioneering validation of domain adaptation of a large language model with 7 billion parameters in TCM domain. We will release both TCMCorpus-1B and TCM-GPT-7B model once accepted to facilitate interdisciplinary development in TCM and NLP, serving as the foundation for further study.

Motivation & Objective

• To address the poor performance of general-purpose LLMs in TCM due to lack of domain-specific knowledge and computational inefficiency.
• To construct a large-scale, high-quality TCM-specific corpus to enable effective pre-training for domain adaptation.
• To develop an efficient fine-tuning strategy that preserves pretrained weights while adapting to TCM-specific tasks.
• To validate the effectiveness of domain-adaptive pre-training on real-world TCM NLP tasks such as diagnosis and examination.
• To release TCM-Corpus-1B and TCM-GPT-7B to support future interdisciplinary research in TCM and NLP.

Proposed method

• Constructed TCM-Corpus-1B by identifying TCM keywords and retrieving relevant text from a general-domain corpus.
• Applied LoRA (Low-Rank Adaptation) to fine-tune the LLM, freezing the original model weights and only training low-rank projection matrices.
• Used the LoRA technique to efficiently adapt the model to TCM tasks with minimal additional parameters.
• Pre-trained and fine-tuned the TCM-GPT-7B model on the TCM-Corpus-1B using the LoRA-adapted training procedure.
• Evaluated the model on two TCM-specific tasks: TCM examination (multiple-choice) and TCM diagnosis (clinical reasoning).
• Released both TCM-Corpus-1B and TCM-GPT-7B to support open research and reproducibility.

Figure 1: The overview of TCMDA

Experimental results

Research questions

• RQ1 Can domain-specific pre-training significantly improve the performance of large language models on TCM NLP tasks?
• RQ2 How effective is LoRA-based fine-tuning for adapting a general LLM to the TCM domain with limited labeled data?
• RQ3 What is the impact of a large-scale, curated TCM corpus on model generalization and factual consistency in TCM reasoning?
• RQ4 Can a 7-billion-parameter LLM achieve state-of-the-art performance in TCM-related tasks after domain adaptation?
• RQ5 How does domain-adapted TCM-GPT-7B compare to random or non-adapted models in generating accurate and meaningful clinical responses?

Key findings

• TCM-GPT-7B achieved a 17% relative accuracy improvement on the TCM-EXAM dataset compared to baseline models.
• On the TCM diagnosis task, TCM-GPT-7B showed a 12% relative accuracy gain over competing models.
• The model’s response in the case study correctly identified '解索脉' (untwining rope pulse) as the answer, while the random baseline incorrectly selected '鱼翔脉' (waving fish pulse).
• The LoRA-based adaptation method enabled efficient fine-tuning with minimal parameter updates, preserving the original model’s knowledge while adapting to TCM-specific terminology and logic.
• The TCM-Corpus-1B corpus significantly enhanced the model’s ability to generate factually grounded and contextually appropriate responses in TCM clinical reasoning.
• To the best of the authors’ knowledge, this is the first successful domain adaptation of a 7B-parameter LLM in the TCM domain, validated on real-world tasks.

Figure 2: The Impact of Corpus Size on Performance