[Paper Review] Bridging the Skills Gap: Evaluating an AI-Assisted Provider Platform to Support Care Providers with Empathetic Delivery of Protocolized Therapy

William R. Kearns, Jessica Bertram+6 more|PubMed|Jan 8, 2024

Digital Mental Health InterventionsPsychology7 references3 citations

TL;DR

This study evaluates the AI-Assisted Provider Platform (A2P2), a text-based virtual therapy interface that uses AI to suggest empathetic, protocol-compliant responses during therapy sessions. It significantly reduced response times by 29.34%, tripled empathic response accuracy (p=0.0001), and improved goal recommendation accuracy by 66.67% across providers with and without mental health expertise, demonstrating strong usability and effectiveness in bridging clinical skills gaps.

ABSTRACT

Despite the high prevalence and burden of mental health conditions, there is a global shortage of mental health providers. Artificial Intelligence (AI) methods have been proposed as a way to address this shortage, by supporting providers with less extensive training as they deliver care. To this end, we developed the AI-Assisted Provider Platform (A2P2), a text-based virtual therapy interface that includes a response suggestion feature, which supports providers in delivering protocolized therapies empathetically. We studied providers with and without expertise in mental health treatment delivering a therapy session using the platform with (intervention) and without (control) AI-assistance features. Upon evaluation, the AI-assisted system significantly decreased response times by 29.34% (p=0.002), tripled empathic response accuracy (p=0.0001), and increased goal recommendation accuracy by 66.67% (p=0.001) across both user groups compared to the control. Both groups rated the system as having excellent usability.

Motivation & Objective

• Address the global shortage of mental health providers by leveraging AI to support less-experienced clinicians.
• Develop a text-based AI platform that enhances empathetic delivery of protocolized therapy.
• Evaluate whether AI assistance improves response quality, speed, and accuracy across providers with varying levels of clinical expertise.
• Assess usability and perceived value of the AI system among non-specialist and expert providers.
• Determine if AI support can effectively close the clinical skills gap in mental health care delivery.

Proposed method

• Design a text-based virtual therapy interface with real-time AI response suggestions for protocolized therapy.
• Implement a retrieval-augmented generation (RAG) pipeline to ground AI responses in clinical guidelines and empathetic communication principles.
• Integrate a fine-tuned LLM to generate contextually appropriate, empathetic, and protocol-compliant responses.
• Conduct a controlled study comparing AI-assisted and non-assisted therapy sessions across two provider groups: mental health experts and non-experts.
• Use automated metrics to evaluate response time, empathic accuracy, and goal recommendation accuracy.
• Apply usability assessments (e.g., System Usability Scale) to evaluate user experience across both groups.

Experimental results

Research questions

• RQ1 Does AI assistance reduce response time during protocolized therapy sessions for both expert and non-expert providers?
• RQ2 To what extent does AI assistance improve the accuracy of empathetic responses in therapy delivery?
• RQ3 How does AI assistance affect the accuracy of goal-related recommendations in therapy sessions?
• RQ4 Is the AI platform perceived as usable and valuable by providers with varying levels of clinical training?
• RQ5 Can AI support effectively narrow the clinical skills gap between expert and non-expert providers in mental health care?

Key findings

• The AI-assisted system reduced provider response times by 29.34% compared to the control condition (p=0.002).
• Empathic response accuracy improved threefold in the AI-assisted condition (p=0.0001), regardless of provider expertise level.
• Goal recommendation accuracy increased by 66.67% when using AI assistance (p=0.001), indicating stronger alignment with treatment protocols.
• Both expert and non-expert providers rated the system as having excellent usability, with no significant difference in perceived ease of use.
• The AI system demonstrated consistent performance gains across provider groups, suggesting it effectively supports less-experienced clinicians in delivering high-quality, empathetic care.