[Paper Review] IvyGPT: InteractiVe Chinese pathwaY language model in medical domain

Rongsheng Wang, Yaofei Duan+7 more|arXiv (Cornell University)|Jul 20, 2023

Topic Modeling4 citations

TL;DR

IvyGPT is a 33-billion-parameter Chinese medical language model fine-tuned via supervised fine-tuning and Reinforcement Learning from Human Feedback (RLHF) using high-quality, real doctor-patient QA data. Trained with QLoRA for low-resource efficiency, it achieves state-of-the-art semantic similarity (93.58) to human doctor responses, outperforming existing models in response richness and medical accuracy.

ABSTRACT

General large language models (LLMs) such as ChatGPT have shown remarkable success. However, such LLMs have not been widely adopted for medical purposes, due to poor accuracy and inability to provide medical advice. We propose IvyGPT, an LLM based on LLaMA that is trained and fine-tuned with high-quality medical question-answer (QA) instances and Reinforcement Learning from Human Feedback (RLHF). After supervised fine-tuning, IvyGPT has good multi-turn conversation capabilities, but it cannot perform like a doctor in other aspects, such as comprehensive diagnosis. Through RLHF, IvyGPT can output richer diagnosis and treatment answers that are closer to human. In the training, we used QLoRA to train 33 billion parameters on a small number of NVIDIA A100 (80GB) GPUs. Experimental results show that IvyGPT has outperformed other medical GPT models.

Motivation & Objective

• To develop a medical LLM tailored for the Chinese healthcare context, addressing limitations of general-purpose models in clinical accuracy and cultural relevance.
• To improve response quality in medical consultations by integrating high-fidelity, real-world doctor-patient conversation data.
• To enable effective training of large medical models (33B parameters) on limited hardware using quantized low-rank adaptation (QLoRA).
• To enhance model responses through Reinforcement Learning from Human Feedback (RLHF), making outputs more comprehensive and clinically aligned.
• To establish a benchmark for semantic similarity between AI-generated and human doctor responses in the Chinese medical domain.

Proposed method

• The model is based on LLaMA and fine-tuned using supervised fine-tuning (SFT) on a curated dataset of 307,038 high-quality Chinese medical QA pairs.
• A reward model is trained using human feedback to evaluate response quality, with preference data collected from annotators rating model outputs.
• Reinforcement Learning from Human Feedback (RLHF) is applied to optimize response generation, improving depth and clinical relevance.
• QLoRA is employed to fine-tune the 33B-parameter model with 4-bit quantization, enabling training on a single 80GB A100 GPU.
• The training pipeline includes data filtering via GPT-4 to correct grammatical and common-sense errors, ensuring data quality.
• Semantic similarity is evaluated using word2vec embeddings and cosine similarity between model outputs and human doctor responses.

Experimental results

Research questions

• RQ1 Can a large language model fine-tuned on real doctor-patient conversations achieve higher semantic similarity to human medical responses than existing models?
• RQ2 To what extent does RLHF improve the clinical depth and comprehensiveness of medical responses compared to supervised fine-tuning alone?
• RQ3 Can a 33B-parameter medical LLM be effectively trained on limited hardware using QLoRA with 4-bit quantization?
• RQ4 How does the quality and realism of training data impact the model’s ability to generate accurate, context-aware medical responses?
• RQ5 What is the trade-off between response length and information richness in medical LLMs, and how does RLHF affect this?

Key findings

• IvyGPT achieves a semantic similarity score of 93.58 with human doctor responses, significantly outperforming baseline models such as ChatMed (84.51) and MedicalGPT (83.73).
• The model generates responses with an average word count of 271.05, indicating richer information coverage compared to other Chinese medical LLMs.
• Reinforcement learning leads to a substantial increase in response length and depth, demonstrating improved ability to provide comprehensive medical advice.
• QLoRA enables efficient fine-tuning of the 33B-parameter model on a single 80GB A100 GPU, reducing training time and memory footprint compared to full fine-tuning or standard LoRA.
• The model’s performance is robust across diverse medical scenarios, showing strong generalization on multi-turn, complex medical queries.
• The dataset of 307,038 real doctor-patient QA pairs, enhanced with GPT-4 validation, contributes to the model’s high factual accuracy and clinical relevance.