[论文解读] A Computational Framework for Behavioral Assessment of LLM Therapists

Yu Ying Chiu, Ashish Sharma+2 more|arXiv (Cornell University)|Jan 1, 2024

Mental Health via Writing被引用 16

一句话总结

Bolt 是一个框架，用于系统性地刻画 LLM 心理治疗师行为，比较它与高质量和低质量的人类治疗，并探索提示如何影响行为。它使用模拟的 client-LLM 会话和心理治疗技术分类法来识别行为。

ABSTRACT

The emergence of large language models (LLMs) like ChatGPT has increased interest in their use as therapists to address mental health challenges and the widespread lack of access to care. However, experts have emphasized the critical need for systematic evaluation of LLM-based mental health interventions to accurately assess their capabilities and limitations. Here, we propose BOLT, a proof-of-concept computational framework to systematically assess the conversational behavior of LLM therapists. We quantitatively measure LLM behavior across 13 psychotherapeutic approaches with in-context learning methods. Then, we compare the behavior of LLMs against high- and low-quality human therapy. Our analysis based on Motivational Interviewing therapy reveals that LLMs often resemble behaviors more commonly exhibited in low-quality therapy rather than high-quality therapy, such as offering a higher degree of problem-solving advice when clients share emotions. However, unlike low-quality therapy, LLMs reflect significantly more upon clients' needs and strengths. Our findings caution that LLM therapists still require further research for consistent, high-quality care.

研究动机与目标

• 促使对用于精神健康护理的 LLM 进行系统性行为评估的必要性。
• 开发一个计算框架（Bolt），在一系列技术上量化 LLM 心理治疗师的行为。
• 将 LLM 心理治疗师的行为与高质量和低质量的人类治疗进行比较。
• 探索提示和模型选择如何影响行为与高质量治疗的一致性。

提出的方法

• 介绍 Bolt，一个基于系统提示的框架，利用公开的治疗数据集在LLM和模拟客户之间模拟治疗对话。
• 用来自既有心理治疗技术的13种治疗师行为和6种客户行为对话语句进行注释。
• 在多标签和二元标签行为分类任务上评估 GPT-3、GPT-4 系列模型，以及 Llama2 变体。
• 使用基于情景的学习，结合心理治疗定义和示例来识别行为；与高质量和低质量人类治疗基线进行比较。
• 分析行为的频率、时间序列顺序及跨模型的适应性；评估显式提示变体对行为的影响。

实验结果

研究问题

• RQ1 Can Bolt reliably identify therapist and client behaviors from therapy conversations?
• RQ2 How do LLM therapist behaviors compare to high- and low-quality human therapy sessions?
• RQ3 Do prompting strategies and model choice steer LLMs toward higher-quality therapeutic behaviors?
• RQ4 Are LLMs more prone to problem-solving/solutions or reflective/normalize behaviors compared with humans?
• RQ5 To what extent can LLMs reflect client needs and strengths similarly to high-quality therapy?

主要发现

• Prompting with psychotherapy definitions and examples yields the best macro-F1 for therapist behavior (57.7% macro-F1).
• Client behavior classification with prompting (binary-label) achieves the best macro-F1 (76.7%).
• LLM therapists show higher problem-solving behavior similar to low-quality human therapy, but also reflect client emotions and experiences more than typical low-quality therapy.
• GPT-4 and GPT-3.5-turbo generally exhibit more solution-focused behavior than Llama2 variants, suggesting RLHF-aligned tendencies influence these patterns.
• Simulated LLM therapy often aligns more with low-quality human therapy in behavior frequencies, indicating current non-ideal alignment with high-quality care.