[Paper Review] Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients

HyoJe Jung, Yunha Kim+11 more|arXiv (Cornell University)|Apr 8, 2024

Electronic Health Records Systems7 citations

TL;DR

The paper fine-tunes various open-source large language models on cardiology progress notes to generate discharge notes, with Mistral-7B achieving strong qualitative and competitive quantitative performance for clinical documentation.

ABSTRACT

Medical documentation, including discharge notes, is crucial for ensuring patient care quality, continuity, and effective medical communication. However, the manual creation of these documents is not only time-consuming but also prone to inconsistencies and potential errors. The automation of this documentation process using artificial intelligence (AI) represents a promising area of innovation in healthcare. This study directly addresses the inefficiencies and inaccuracies in creating discharge notes manually, particularly for cardiac patients, by employing AI techniques, specifically large language model (LLM). Utilizing a substantial dataset from a cardiology center, encompassing wide-ranging medical records and physician assessments, our research evaluates the capability of LLM to enhance the documentation process. Among the various models assessed, Mistral-7B distinguished itself by accurately generating discharge notes that significantly improve both documentation efficiency and the continuity of care for patients. These notes underwent rigorous qualitative evaluation by medical expert, receiving high marks for their clinical relevance, completeness, readability, and contribution to informed decision-making and care planning. Coupled with quantitative analyses, these results confirm Mistral-7B's efficacy in distilling complex medical information into concise, coherent summaries. Overall, our findings illuminate the considerable promise of specialized LLM, such as Mistral-7B, in refining healthcare documentation workflows and advancing patient care. This study lays the groundwork for further integrating advanced AI technologies in healthcare, demonstrating their potential to revolutionize patient documentation and support better care outcomes.

Motivation & Objective

• Reduce manual effort and improve accuracy in discharge note generation for cardiac patients.
• Evaluate multiple open-source LLMs on a cardiology discharge note task using supervised fine-tuning and PEFT.
• Assess both quantitative and expert qualitative performance to determine clinical utility.

Proposed method

• Use Progress Notes as input and Discharge Notes as target labels for training.
• Fine-tune multiple pre-trained LLMs with SFT using PEFT (QLoRA) to adapt to medical documentation.
• Apply Unsloth to optimize fine-tuning efficiency and LoRA-based parameter updates.
• Evaluate models with ROUGE, BLEU, BERT Score, Perplexity, and expert-cardio qualitative scores.

Experimental results

Research questions

• RQ1 Can specialized LLMs generate clinically accurate and actionable discharge notes from cardiology progress notes?
• RQ2 Which open-source LLMs (and fine-tuning strategies) best balance accuracy, completeness, readability, and utility in discharge note generation?
• RQ3 How do quantitative metrics align with expert qualitative assessments in medical document generation?

Key findings

• Mistral-7B achieves high qualitative scores and favorable quantitative metrics among tested models.
• Different models excel on different quantitative metrics, indicating no single best model across all criteria.
• Qualitative expert evaluation shows strong accuracy, completeness, readability, consistency, and utility for Mistral-7B.
• The study demonstrates potential for AI-driven discharge note automation to enhance documentation efficiency and continuity of care in cardiology.
• Limitations include domain-specific evaluation gaps and data standardization challenges for progress notes.