[Paper Review] PATIENT-Ψ: Using Large Language Models to Simulate Patients for Training Mental Health Professionals

Ruiyi Wang, Stephanie Milani+10 more|arXiv (Cornell University)|May 30, 2024

Digital Mental Health Interventions5 citations

TL;DR

PATIENT-Ψ combines CBT cognitive models with LLMs to simulate patient interactions for training mental health professionals, and introduces an interactive trainer to practice cognitive model formulation.

ABSTRACT

Mental illness remains one of the most critical public health issues. Despite its importance, many mental health professionals highlight a disconnect between their training and actual real-world patient practice. To help bridge this gap, we propose PATIENT-Ψ, a novel patient simulation framework for cognitive behavior therapy (CBT) training. To build PATIENT-Ψ, we construct diverse patient cognitive models based on CBT principles and use large language models (LLMs) programmed with these cognitive models to act as a simulated therapy patient. We propose an interactive training scheme, PATIENT-Ψ-TRAINER, for mental health trainees to practice a key skill in CBT -- formulating the cognitive model of the patient -- through role-playing a therapy session with PATIENT-Ψ. To evaluate PATIENT-Ψ, we conducted a comprehensive user study of 13 mental health trainees and 20 experts. The results demonstrate that practice using PATIENT-Ψ-TRAINER enhances the perceived skill acquisition and confidence of the trainees beyond existing forms of training such as textbooks, videos, and role-play with non-patients. Based on the experts' perceptions, PATIENT-Ψ is perceived to be closer to real patient interactions than GPT-4, and PATIENT-Ψ-TRAINER holds strong promise to improve trainee competencies. Our code and data are released at \url{https://github.com/ruiyiw/patient-psi}.

Motivation & Objective

• Bridge the gap between training and real patient interactions in mental health care.
• Create diverse, expert-verified cognitive models (CCDs) of patients with depression/anxiety.
• Integrate CCDs with LLMs to generate authentic simulated patient behavior.
• Provide an interactive trainer that lets learners formulate a patient’s cognitive model and receive feedback.

Proposed method

• Develop CCDs from CBT theory and expert input to represent core beliefs, automatic thoughts, emotions, and behaviors.
• Program an LLM with patient cognitive models and six predefined relational styles to simulate patient dialogue.
• Create Patient-Ψ by conditioning the LLM on CCDs and style instructions to perform CBT-based role-plays.
• Design Patient-Ψ-Trainer as a web-based interactive tool where trainees practice model formulation and compare with reference CCDs.
• Evaluate fidelity and training effectiveness via expert and trainee user studies comparing Patient-Ψ-Trainer to a GPT-4 baseline.

Experimental results

Research questions

• RQ1 How faithfully can a CCD-informed LLM simulate real patient communication across diverse relational styles?
• RQ2 Does training with Patient-Ψ-Trainer improve CBT formulation skills, perceived competence, and confidence more than baseline methods?
• RQ3 Is Patient-Ψ-Trainer preferred over existing CBT training techniques by experts and trainees?
• RQ4 What factors influence the perceived usefulness and adoption of a new AI-powered patient simulation tool?
• RQ5 Can this framework generalize beyond CBT to other mental health training paradigms?

Key findings

• Experts rated Patient-Ψ as more faithful to real patients than GPT-4 on symptoms, cognitions, emotions, and relational styles (median 4.0–4.5).
• Experts rated Patient-Ψ-Trainer highly for improving CBT formulation skills and identifying beliefs and maladaptive thinking (median up to 5.0).
• Experts preferred Patient-Ψ-Trainer over GPT-4 baseline for overall CBT skills and adoption in classrooms (median 5.0 for adoption).
• Trainees showed stronger preference for Patient-Ψ-Trainer (median 4.5) and reported greater perceived skill and confidence improvements than baseline methods.
• Across dimensions of CCD accuracy, about 75% of simulated patients were very to extremely accurate overall; 75–90% were very to extremely accurate on specific dimensions.