[Paper Review] DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task

Honglin Xiong, Sheng Wang+6 more|arXiv (Cornell University)|Apr 3, 2023

Artificial Intelligence in Healthcare and Education71 citations

TL;DR

The paper demonstrates fine-tuning a bilingual Chinese medical dialogue model (ChatGLM-6B) into DoctorGLM using LoRA and other efficiency techniques, achieving affordable healthcare-domain fine-tuning on modest hardware. It provides a low-cost pipeline for multilingual medical LLMs and shares practical results and limitations.

ABSTRACT

The recent progress of large language models (LLMs), including ChatGPT and GPT-4, in comprehending and responding to human instructions has been remarkable. Nevertheless, these models typically perform better in English and have not been explicitly trained for the medical domain, resulting in suboptimal precision in diagnoses, drug recommendations, and other medical advice. Additionally, training and deploying a dialogue model is still believed to be impossible for hospitals, hindering the promotion of LLMs. To tackle these challenges, we have collected databases of medical dialogues in Chinese with ChatGPT's help and adopted several techniques to train an easy-deploy LLM. Remarkably, we were able to fine-tune the ChatGLM-6B on a single A100 80G in 13 hours, which means having a healthcare-purpose LLM can be very affordable. DoctorGLM is currently an early-stage engineering attempt and contain various mistakes. We are sharing it with the broader community to invite feedback and suggestions to improve its healthcare-focused capabilities: https://github.com/xionghonglin/DoctorGLM.

Motivation & Objective

• Motivate development of language models tailored to medical domains and non-English languages.
• Describe a low-cost, end-to-end pipeline for fine-tuning Chinese medical dialogue models.
• Showcase techniques enabling healthcare-focused LLMs on affordable hardware.

Proposed method

• Fine-tune ChatGLM-6B with LoRA on Chinese medical dialogue data using an A100 80G GPU.
• Translate English medical datasets to Chinese via ChatGPT and distill with a BART-based model.
• Incorporate a prompt-design module using disease knowledge (Merck Manual-derived) to guide responses.
• Compare LoRA with P-tuning V2 for parameter-efficient fine-tuning.
• Evaluate generation with top-p and temperature settings to manage output diversity.

Figure 1 : Overview of DoctorGLM fine-tuning and inference pipeline.

Experimental results

Research questions

• RQ1 Can a healthcare-focused LLM be effectively fine-tuned in Chinese using resource-efficient methods?
• RQ2 What is the cost, in hardware and time, to train a Chinese medical dialogue model on in-house data?
• RQ3 How do LoRA and P-tuning V2 compare for fine-tuning efficiency and performance in the medical domain?
• RQ4 What role does a prompt-design module play in improving reliability and accuracy of medical responses?
• RQ5 What are the practical limitations and deployment considerations for hospital-scale use of such models?

Key findings

• Fine-tuning DoctorGLM on Chinese medical dialogues is achievable on a single A100 80G GPU in 13 hours using LoRA.
• The cost to fine-tune on 100,000 QA pairs is reported to be approximately 18.75 USD under the described setup.
• Inference can be performed on consumer GPUs with about 13 GB memory, though deployment constraints exist.
• LoRA and P-tuning V2 offer comparable performance with different trade-offs in parameter efficiency.
• The authors acknowledge multiple technical limitations and emphasize this as an early-stage engineering effort.

Figure 2 : The implementation of large-scale translation. A tiny and high-quality dataset is built through ChatGPT. The collected dataset serves as a fine-tuning set for a pre-trained language model, enabling it to perform specialized machine translation.