[Paper Review] Interactive Agents: Simulating Counselor-Client Psychological Counseling via Role-Playing LLM-to-LLM Interactions

Huachuan Qiu, Zhenzhong Lan|arXiv (Cornell University)|Aug 28, 2024

Artificial Intelligence in Law4 citations

TL;DR

This paper proposes a role-playing LLM-to-LLM framework simulating counselor-client psychological counseling using two GPT-4 models: one as a client with a real-life profile and one as an experienced counselor using integrative therapy. The method generates synthetic, privacy-preserving dialogues that closely mirror human interactions, demonstrating strong performance in dialogue quality and counselor behavior similarity, offering a scalable alternative to human-annotated data.

ABSTRACT

Virtual counselors powered by large language models (LLMs) aim to create interactive support systems that effectively assist clients struggling with mental health challenges. To replicate counselor-client conversations, researchers have built an online mental health platform that allows professional counselors to provide clients with text-based counseling services for about an hour per session. Notwithstanding its effectiveness, challenges exist as human annotation is time-consuming, cost-intensive, privacy-protected, and not scalable. To address this issue and investigate the applicability of LLMs in psychological counseling conversation simulation, we propose a framework that employs two LLMs via role-playing for simulating counselor-client interactions. Our framework involves two LLMs, one acting as a client equipped with a specific and real-life user profile and the other playing the role of an experienced counselor, generating professional responses using integrative therapy techniques. We implement both the counselor and the client by zero-shot prompting the GPT-4 model. In order to assess the effectiveness of LLMs in simulating counselor-client interactions and understand the disparities between LLM- and human-generated conversations, we evaluate the synthetic data from various perspectives. We begin by assessing the client's performance through automatic evaluations. Next, we analyze and compare the disparities between dialogues generated by the LLM and those generated by professional counselors. Furthermore, we conduct extensive experiments to thoroughly examine the performance of our LLM-based counselor trained with synthetic interactive dialogues by benchmarking against state-of-the-art models for mental health.

Motivation & Objective

• To address the scalability, cost, and privacy limitations of human-annotated mental health counseling data.
• To simulate realistic, interactive counselor-client dialogues without relying on real human participants.
• To evaluate the quality and therapeutic fidelity of LLM-generated dialogues compared to human-generated sessions.
• To benchmark an LLM-based counselor trained on synthetic data against state-of-the-art mental health dialogue models.
• To explore ethical and practical challenges in using LLMs for psychological counseling simulation.

Proposed method

• Two GPT-4 models are used in a zero-shot role-playing setup: one as a client with a predefined, real-life user profile, and one as a counselor trained in integrative therapy techniques.
• The client model is prompted to simulate a specific mental health issue with personal, contextual details to enhance realism.
• The counselor model generates responses using integrative therapy approaches, focusing on exploration, insight, and action planning.
• Synthetic dialogues are generated through iterative, back-and-forth interactions mimicking real sessions.
• The framework enables automatic evaluation of dialogue quality and comparison with human-generated sessions using multiple metrics.
• Retrieval-augmented generation (RAG) and prompt optimization are proposed for future improvements to enhance realism and verisimilitude.

Experimental results

Research questions

• RQ1 RQ1: How can LLMs effectively and automatically simulate counselor-client interactions through role-playing?
• RQ2 RQ2: How do LLM-generated dialogues compare in quality and therapeutic fidelity to those produced by professional human counselors?
• RQ3 RQ3: How well does an LLM-based counselor, fine-tuned on synthetic data, perform compared to state-of-the-art mental health dialogue models?
• RQ4 RQ4: What are the ethical risks and limitations of using LLMs to simulate both clients and counselors in psychological counseling?
• RQ5 RQ5: How do client resistance and session progression affect the quality and depth of LLM-generated dialogues?

Key findings

• The LLM-based counselor generates dialogues that are highly similar in structure and therapeutic quality to those produced by professional human counselors, as measured by automatic evaluation metrics.
• The synthetic dialogues exhibit strong coherence, emotional engagement, and appropriate use of integrative therapy techniques, indicating high realism.
• The LLM-based counselor, when fine-tuned on synthetic data, achieves competitive performance against state-of-the-art models in mental health dialogue benchmarks.
• The role-playing framework successfully simulates a full counseling session with a defined client profile, demonstrating scalability and privacy preservation.
• Ethical concerns such as inappropriate advice, oversimplified client modeling, and limited counseling depth were identified as key limitations in LLM simulations.
• The study confirms that LLMs can generate high-quality, privacy-free counseling dialogues, offering a viable alternative to human-annotated data collection.