[Paper Review] Large Language Models for Medical OSCE Assessment: A Novel Approach to Transcript Analysis

Ameer Hamza Shakur, Michael J. Holcomb+6 more|arXiv (Cornell University)|Oct 11, 2024

Global Security and Public Health4 citations

TL;DR

This study proposes using large language models (LLMs) to automate assessment of medical students' communication skills in Objective Structured Clinical Examinations (OSCEs), specifically evaluating whether students summarize patient medical histories from transcribed exam interviews. Using Whisper-v3 for ASR and frontier LLMs like GPT-4 with chain-of-thought and retrieval-augmented prompting, the approach achieved high agreement (Cohen’s kappa = 0.88) with human graders, demonstrating strong potential for scalable, cost-effective OSCE grading with actionable feedback.

ABSTRACT

Grading Objective Structured Clinical Examinations (OSCEs) is a time-consuming and expensive process, traditionally requiring extensive manual effort from human experts. In this study, we explore the potential of Large Language Models (LLMs) to assess skills related to medical student communication. We analyzed 2,027 video-recorded OSCE examinations from the University of Texas Southwestern Medical Center (UTSW), spanning four years (2019-2022), and several different medical cases or "stations." Specifically, our focus was on evaluating students' ability to summarize patients' medical history: we targeted the rubric item 'did the student summarize the patients' medical history?' from the communication skills rubric. After transcribing speech audio captured by OSCE videos using Whisper-v3, we studied the performance of various LLM-based approaches for grading students on this summarization task based on their examination transcripts. Using various frontier-level open-source and proprietary LLMs, we evaluated different techniques such as zero-shot chain-of-thought prompting, retrieval augmented generation, and multi-model ensemble methods. Our results show that frontier LLM models like GPT-4 achieved remarkable alignment with human graders, demonstrating a Cohen's kappa agreement of 0.88 and indicating strong potential for LLM-based OSCE grading to augment the current grading process. Open-source models also showed promising results, suggesting potential for widespread, cost-effective deployment. Further, we present a failure analysis identifying conditions where LLM grading may be less reliable in this context and recommend best practices for deploying LLMs in medical education settings.

Motivation & Objective

• To address the time-consuming and expensive nature of manual OSCE grading by human experts in medical education.
• To investigate whether large language models (LLMs) can reliably assess medical students’ communication skills, particularly history summarization, from transcribed OSCE interviews.
• To evaluate the performance of various LLM architectures and prompting strategies in aligning with human graders on a standardized rubric.
• To identify failure modes and ethical risks in deploying LLMs for medical assessment, and to recommend best practices for hybrid human-AI grading workflows.

Proposed method

• Transcribed 2,027 video-recorded OSCE encounters from 2019–2022 using Whisper-v3, a state-of-the-art automatic speech recognition model.
• Focused on the communication skills rubric item: 'Did the student summarize the patient’s medical history?'
• Applied multiple LLM prompting techniques, including zero-shot chain-of-thought, retrieval-augmented generation, and multi-model ensemble methods.
• Evaluated both proprietary models (e.g., GPT-4, GPT-3.5) and open-source models (e.g., Mixtral-8x7B, Starling-7B) for grading consistency with human experts.
• Used Cohen’s kappa to measure inter-annotator agreement between LLM outputs and human graders, ensuring reliability assessment.
• Conducted failure analysis to identify common LLM errors such as hallucination, prompt misinterpretation, and contextual confusion.

Experimental results

Research questions

• RQ1 Can LLMs achieve high agreement with human experts in grading medical students’ history summarization during OSCEs?
• RQ2 How do different LLM architectures and prompting strategies (e.g., chain-of-thought, retrieval-augmented generation) affect grading reliability?
• RQ3 What are the primary failure modes of LLM-based grading in this context, and how can they be mitigated?
• RQ4 To what extent can open-source LLMs replicate the performance of proprietary models in this medical education setting?

Key findings

• GPT-4 achieved a Cohen’s kappa agreement of 0.88 with human graders, indicating substantial to almost perfect agreement on the history summarization task.
• Open-source LLMs such as Mixtral-8x7B and Starling-7B showed promising performance, suggesting potential for cost-effective, scalable deployment in medical education.
• Retrieval-augmented generation and multi-model ensemble methods reduced error rates, particularly by minimizing hallucination and context confusion.
• The most common LLM errors were hallucination (generating non-existent summaries), prompt misinterpretation (mislabeling symptom discussions as summaries), and contextual confusion (misclassifying diagnostic explanations as summaries).
• Transcription quality issues—such as background noise or recording errors—were a key upstream factor affecting downstream LLM performance, highlighting the need for audio quality validation before AI assessment.
• A hybrid human-AI workflow, where LLMs act as initial graders and human experts review and refine outputs, is recommended to ensure fairness, transparency, and continuous model improvement.