[Paper Review] Improving Clinical Documentation with AI: A Comparative Study of Sporo AI Scribe and GPT-4o mini

Chanseo Lee, Shubham Kumar+2 more|arXiv (Cornell University)|Oct 20, 2024

Artificial Intelligence in Healthcare and EducationMedicine3 citations

TL;DR

This study evaluates Sporo AI Scribe, a multi-agent AI scribe using fine-tuned medical LLMs, against GPT-4o Mini for clinical documentation. Sporo outperformed GPT-4o Mini in clinical content recall, precision, and F1 scores, with higher clinician satisfaction and fewer hallucinations, demonstrating superior accuracy and reliability in automated EHR documentation.

ABSTRACT

AI-powered medical scribes have emerged as a promising solution to alleviate the documentation burden in healthcare. Ambient AI scribes provide real-time transcription and automated data entry into Electronic Health Records (EHRs), with the potential to improve efficiency, reduce costs, and enhance scalability. Despite early success, the accuracy of AI scribes remains critical, as errors can lead to significant clinical consequences. Additionally, AI scribes face challenges in handling the complexity and variability of medical language and ensuring the privacy of sensitive patient data. This case study aims to evaluate Sporo Health's AI scribe, a multi-agent system leveraging fine-tuned medical LLMs, by comparing its performance with OpenAI's GPT-4o Mini on multiple performance metrics. Using a dataset of de-identified patient conversation transcripts, AI-generated summaries were compared to clinician-generated notes (the ground truth) based on clinical content recall, precision, and F1 scores. Evaluations were further supplemented by clinician satisfaction assessments using a modified Physician Documentation Quality Instrument revision 9 (PDQI-9), rated by both a medical student and a physician. The results show that Sporo AI consistently outperformed GPT-4o Mini, achieving higher recall, precision, and overall F1 scores. Moreover, the AI generated summaries provided by Sporo were rated more favorably in terms of accuracy, comprehensiveness, and relevance, with fewer hallucinations. These findings demonstrate that Sporo AI Scribe is an effective and reliable tool for clinical documentation, enhancing clinician workflows while maintaining high standards of privacy and security.

Motivation & Objective

• To assess the performance of Sporo AI Scribe, a multi-agent AI scribe system, against GPT-4o Mini in generating accurate clinical summaries.
• To evaluate the clinical relevance, comprehensiveness, and precision of AI-generated notes compared to clinician-generated ground truth notes.
• To measure clinician satisfaction with AI-generated documentation using a validated instrument (PDQI-9).
• To examine the risk of hallucinations and data privacy in AI scribes within real clinical documentation workflows.
• To determine whether fine-tuned, domain-specific LLMs outperform general-purpose LLMs in clinical documentation tasks.

Proposed method

• Used de-identified patient visit transcripts as input for both Sporo AI Scribe and GPT-4o Mini to generate clinical summaries.
• Evaluated summaries against clinician-generated notes using standard NLP metrics: recall, precision, and F1 score.
• Employed a modified Physician Documentation Quality Instrument revision 9 (PDQI-9) to assess clinician satisfaction with AI-generated notes.
• Applied a multi-agent system architecture in Sporo AI Scribe, leveraging fine-tuned medical language models for improved clinical reasoning.
• Ensured data privacy through de-identification and secure processing, with no exposure of sensitive patient information.
• Conducted dual assessments by a medical student and a physician to ensure reliability in clinician satisfaction ratings.

Experimental results

Research questions

• RQ1 How does Sporo AI Scribe compare to GPT-4o Mini in terms of clinical content recall, precision, and F1 score when summarizing patient visits?
• RQ2 To what extent do clinicians rate Sporo AI Scribe's summaries as more accurate, comprehensive, and relevant than those from GPT-4o Mini?
• RQ3 What is the incidence of hallucinations in summaries generated by Sporo AI Scribe versus GPT-4o Mini?
• RQ4 How does the use of fine-tuned medical LLMs in a multi-agent system affect performance compared to a general-purpose LLM like GPT-4o Mini?
• RQ5 What are the perceived benefits and risks of AI scribes in clinical documentation from a clinician workflow perspective?

Key findings

• Sporo AI Scribe achieved significantly higher recall, precision, and F1 scores compared to GPT-4o Mini in summarizing clinical visit transcripts.
• Clinician assessments using the PDQI-9 rated Sporo AI Scribe's summaries as more accurate, comprehensive, and relevant than those from GPT-4o Mini.
• Sporo AI Scribe generated fewer hallucinations compared to GPT-4o Mini, indicating better factual consistency with clinical content.
• The multi-agent system using fine-tuned medical LLMs in Sporo AI Scribe demonstrated superior performance over the general-purpose GPT-4o Mini model.
• Sporo AI Scribe maintained high standards of privacy and security, with no exposure of sensitive patient data during evaluation.
• The study confirms that domain-specific fine-tuning enhances AI scribe reliability and clinical utility in real-world documentation tasks.