[Paper Review] Healthcare Copilot: Eliciting the Power of General LLMs for Medical Consultation

Zhiyao Ren, Yibing Zhan+3 more|arXiv (Cornell University)|Feb 20, 2024

Medical Coding and Health Information6 citations

TL;DR

Healthcare Copilot enhances general LLMs for medical consultations by orchestrating Dialogue, Memory, and Processing modules and evaluating them with an auto-evaluation scheme using ChatGPT; it improves inquiry, fluency, accuracy, and safety across several backbones.

ABSTRACT

The copilot framework, which aims to enhance and tailor large language models (LLMs) for specific complex tasks without requiring fine-tuning, is gaining increasing attention from the community. In this paper, we introduce the construction of a Healthcare Copilot designed for medical consultation. The proposed Healthcare Copilot comprises three main components: 1) the Dialogue component, responsible for effective and safe patient interactions; 2) the Memory component, storing both current conversation data and historical patient information; and 3) the Processing component, summarizing the entire dialogue and generating reports. To evaluate the proposed Healthcare Copilot, we implement an auto-evaluation scheme using ChatGPT for two roles: as a virtual patient engaging in dialogue with the copilot, and as an evaluator to assess the quality of the dialogue. Extensive results demonstrate that the proposed Healthcare Copilot significantly enhances the capabilities of general LLMs for medical consultations in terms of inquiry capability, conversational fluency, response accuracy, and safety. Furthermore, we conduct ablation studies to highlight the contribution of each individual module in the Healthcare Copilot. Code will be made publicly available on GitHub.

Motivation & Objective

• Motivate leveraging general LLMs for medical consultation without fine-tuning.
• Design a copilot framework that enables safe, multi-round patient interactions.
• Incorporate memory to retain current and historical patient information.
• Provide post-processing reports to summarize consultations for patients and clinicians.

Proposed method

• Introduce a three-component architecture: Dialogue, Memory, and Processing.
• Implement a Function module to categorize tasks (diagnosis, explanation, recommendation) and guide multi-round inquiries.
• Add Safety and Doctor modules to ensure ethics, safety, and professional oversight.
• Memory with Conversation Memory and History Memory to maintain context and summarize history.
• Processing module provides summarization and report generation for consultations.

Experimental results

Research questions

• RQ1 Can a general LLM-based copilot improve medical consultation quality without fine-tuning?
• RQ2 How do Dialogue, Memory, and Processing components contribute to inquiry, fluency, accuracy, and safety?
• RQ3 What is the impact of modular prompts and doctor supervision on real-world medical dialogue?
• RQ4 How effective is auto-evaluation via ChatGPT in assessing medical consultation quality?

Key findings

• Healthcare Copilot significantly improves inquiry capability, conversational fluency, response accuracy, and response safety across backbones like GPT-4, GPT-3.5, LLaMA2, and ChatGLM3.
• Ablation shows the Function, Inquiry, Safety, Conversation Memory, and History Memory modules each contribute to performance, with notable drops when modules are removed.
• GPT-4 generally provides the strongest performance as the backbone for Healthcare Copilot.
• The Safety and Doctor modules enhance ethical compliance and allow professional intervention when needed.
• Open-source medical LLMs (e.g., MedAlpaca-7B) show challenges in adhering to Healthcare Copilot guidelines, highlighting openness as a limiting factor.