[Paper Review] The Impact of AI Assistance on Radiology Reporting: A Pilot Study Using Simulated AI Draft Reports

Julián Acosta, Siddhant Dogra+5 more|arXiv (Cornell University)|Dec 16, 2024

Artificial Intelligence in Healthcare and EducationMedicine3 citations

TL;DR

This pilot study evaluated AI-assisted radiology reporting using simulated GPT-4-generated draft reports on 20 chest CT scans. Radiologists using AI drafts reduced average reporting time by 24% (from 573 to 435 seconds, p=0.003) without a significant increase in clinically significant errors, suggesting AI drafts can enhance efficiency while preserving diagnostic accuracy in real-world workflows.

ABSTRACT

Radiologists face increasing workload pressures amid growing imaging volumes, creating risks of burnout and delayed reporting times. While artificial intelligence (AI) based automated radiology report generation shows promise for reporting workflow optimization, evidence of its real-world impact on clinical accuracy and efficiency remains limited. This study evaluated the effect of draft reports on radiology reporting workflows by conducting a three reader multi-case study comparing standard versus AI-assisted reporting workflows. In both workflows, radiologists reviewed the cases and modified either a standard template (standard workflow) or an AI-generated draft report (AI-assisted workflow) to create the final report. For controlled evaluation, we used GPT-4 to generate simulated AI drafts and deliberately introduced 1-3 errors in half the cases to mimic real AI system performance. The AI-assisted workflow significantly reduced average reporting time from 573 to 435 seconds (p=0.003), without a statistically significant difference in clinically significant errors between workflows. These findings suggest that AI-generated drafts can meaningfully accelerate radiology reporting while maintaining diagnostic accuracy, offering a practical solution to address mounting workload challenges in clinical practice.

Motivation & Objective

• To evaluate the impact of AI-generated draft reports on radiology reporting efficiency and diagnostic accuracy in a real-world workflow simulation.
• To assess whether radiologists can maintain diagnostic accuracy when using AI drafts containing 1–3 intentional errors, simulating real AI system limitations.
• To examine radiologists’ cognitive load and usability perceptions when integrating AI-assisted reporting into their clinical workflow.
• To identify individual variability in response to AI assistance, particularly related to experience level and workflow preferences.
• To provide preliminary evidence supporting the feasibility and benefits of AI-assisted reporting for reducing radiologist workload and burnout.

Proposed method

• Conducted a three-reader, multi-case crossover study using 20 chest CT scans from the CT-RATE dataset, matched by age, sex, and complexity proxies.
• Generated simulated AI draft reports using GPT-4, applying standard negative templates and injecting 1–3 clinically relevant errors in half the cases to mimic real AI performance.
• Implemented a custom web platform with Flask, PostgreSQL, and JavaScript/CSS to host the reading workflow, including worklist and report editing interfaces.
• Used a crossover design where each radiologist alternated between standard template-based and AI-assisted reporting workflows to control for order effects and individual variability.
• Collected reporting times, clinically significant errors, and user perception data via surveys assessing usability, mental effort, and recommendation likelihood.
• Performed statistical analysis comparing reporting times and error rates between standard and AI-assisted workflows, with subgroup analysis by error count.

Figure 1 : Study overview. (A) Reader assignment and crossover between AI-assisted and unassisted workflows. (B) Unassisted workflow using standard template. (C) AI-assisted workflow with pre-generated drafts.

Experimental results

Research questions

• RQ1 Does AI-assisted reporting reduce radiology reporting time compared to standard template-based reporting in a simulated clinical workflow?
• RQ2 Does the presence of intentional AI-generated errors in draft reports lead to a statistically significant increase in clinically significant diagnostic errors?
• RQ3 How do radiologists perceive the usability and cognitive load of AI-assisted reporting compared to standard template use?
• RQ4 To what extent does individual variability in radiologist experience or workflow style affect the efficiency and accuracy gains from AI assistance?
• RQ5 What is the relationship between AI draft quality (e.g., error frequency) and radiologist performance in detecting and correcting errors?

Key findings

• The AI-assisted workflow reduced average reporting time from 573 seconds to 435 seconds, representing a 24% reduction (p=0.003).
• There was no statistically significant difference in clinically significant errors between the standard template (mean 0.38, SD 0.78) and AI-draft (mean 0.27, SD 0.52) workflows.
• All three readers reported the AI-assisted system as easy to use, with unanimous agreement that it would integrate well into clinical workflows.
• Two of three readers reported reduced mental effort with AI assistance, and one reported significantly reduced effort.
• Recommendation scores varied (5, 9, 10 out of 10), indicating mixed enthusiasm despite strong usability feedback.
• Exploratory subgroup analysis showed no significant difference in error rates between AI drafts with or without introduced errors, suggesting radiologists effectively detected and corrected flaws.

Figure 2 : Reporting platform. Top: normal negative template case. Bottom: AI-drafted case.