[Paper Review] Intelligent Documentation in Medical Education: Can AI Replace Manual Case Logging?

Nafiz Imtiaz Khan, Kylie Cleland+2 more|arXiv (Cornell University)|Jan 19, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

This study tests whether large language models can automatically extract procedural radiology cases from reports to replace manual case logging, comparing local and commercial models and prompting strategies.

ABSTRACT

Procedural case logs are a core requirement in radiology training, yet they are time-consuming to complete and prone to inconsistency when authored manually. This study investigates whether large language models (LLMs) can automate procedural case log documentation directly from free-text radiology reports. We evaluate multiple local and commercial LLMs under instruction-based and chain-of-thought prompting to extract structured procedural information from 414 curated interventional radiology reports authored by nine residents between 2018 and 2024. Model performance is assessed using sensitivity, specificity, and F1-score, alongside inference latency and token efficiency to estimate operational cost. Results show that both local and commercial models achieve strong extraction performance, with best F1-scores approaching 0.87, while exhibiting different trade-offs between speed and cost. Automation using LLMs has the potential to substantially reduce clerical burden for trainees and improve consistency in case logging. These findings demonstrate the feasibility of AI-assisted documentation in medical education and highlight the need for further validation across institutions and clinical workflows.

Motivation & Objective

• Assess feasibility of using LLMs to automate radiology procedural case logs from narrative reports.
• Identify procedure categories that are difficult for AI-based extraction.
• Evaluate integration considerations, including latency and cost, for clinical workflow deployment.
• Provide guidance on prompting strategies and model selection for scalable documentation automation.

Proposed method

• Retrospective dataset of 414 radiology reports from nine IR residents (2018–2024) with 39 predefined procedures.
• Annotators established ground truth with Cohen’s Kappa = 0.896.
• Evaluate six models (five open/local, one commercial) in zero-shot, with Instruction Prompting and Chain-of-Thought prompting.
• Metrics: sensitivity, specificity, F1-score, inference time, token usage, and cost estimates.
• Crosswalk benchmark used as a metadata-based baseline for comparison.

Experimental results

Research questions

• RQ1 Can LLMs extract structured procedural data from radiology reports with high accuracy?
• RQ2 Does model performance vary by procedure category (vascular diagnosis, vascular intervention, non-vascular intervention)?
• RQ3 How do local vs. commercial models compare in speed, cost, and accuracy under different prompting strategies?
• RQ4 What are practical deployment considerations (latency, token usage, and cost) for real-world workflow integration?

Key findings

• Commercial model Claude-3.5-Haiku with Chain-of-Thought prompting achieves the highest F1-score (86.89%) among all configurations.
• Local model Qwen-2.5:72B with Chain-of-Thought prompting attains an F1-score of 86.66%, with high sensitivity and specificity.
• Crosswalk metadata baseline shows high specificity (99.40%) but lower sensitivity (65.46%), underlining its limitation in free-text extraction.
• Prompting strategy (CoT) generally improves F1-scores and reduces false positives across models, especially for complex categories like Vascular Interventions.
• Inference time varies by model: Claude-3.5-Haiku IP ~1.97s per procedure (fastest); Qwen-2.5:72B CoT ~13.47s per procedure (slower but accurate).
• Estimated per-resident annual time savings exceed 35 hours when automated logging replaces manual entry.