[Paper Review] ChatCounselor: A Large Language Models for Mental Health Support

June M. Liu, Donghao Li+4 more|arXiv (Cornell University)|Sep 27, 2023

Digital Mental Health Interventions36 citations

TL;DR

ChatCounselor is an LLM fine-tuned on professional counseling data (Psych8k) to provide mental health support and surpasses open-source models, approaching ChatGPT performance.

ABSTRACT

This paper presents ChatCounselor, a large language model (LLM) solution designed to provide mental health support. Unlike generic chatbots, ChatCounselor is distinguished by its foundation in real conversations between consulting clients and professional psychologists, enabling it to possess specialized knowledge and counseling skills in the field of psychology. The training dataset, Psych8k, was constructed from 260 in-depth interviews, each spanning an hour. To assess the quality of counseling responses, the counseling Bench was devised. Leveraging GPT-4 and meticulously crafted prompts based on seven metrics of psychological counseling assessment, the model underwent evaluation using a set of real-world counseling questions. Impressively, ChatCounselor surpasses existing open-source models in the counseling Bench and approaches the performance level of ChatGPT, showcasing the remarkable enhancement in model capability attained through high-quality domain-specific data.

Motivation & Objective

• Construct a high-quality counseling dataset (Psych8k) from licensed counselors.
• Develop a domain-specific instruction-tuning pipeline for psychological counseling.
• Evaluate counseling quality using a dedicated Counseling Bench and GPT-4 automated evaluation.
• Demonstrate that domain-focused data improves performance over open-source baselines.
• Make code available to reproduce and extend the work.

Proposed method

• Assemble Psych8k from 260 real-life counseling interviews, distill 8,187 instruction pairs totaling ~2M tokens using GPT-4.
• Fine-tune a LLaMA-7B base model with domain-specific instructions for counseling tasks.
• Use instruction tuning with auto-regressive training to align responses with counseling skills.
• Evaluate using a seven-m Strategy-based Counseling Bench and 229 real-world questions.
• Leverage GPT-4 as an automated referee to score model outputs across strategies.
• Report training setup details including context length, batch size, optimizer, and hardware.

Experimental results

Research questions

• RQ1 Can domain-specific counseling data improve LLM performance in mental health support?
• RQ2 How does ChatCounselor perform on domain-relevant counseling strategies compared to open-source baselines?
• RQ3 Does a small amount of high-quality, professional data yield measurable gains in counseling quality?
• RQ4 Can automated GPT-4 evaluation provide robust, rapid assessment of counseling capabilities?

Key findings

• ChatCounselor outperforms open-source LLaMA-7B, Alpaca-7B, ChatGLM-v2-7B, and Robins-v2-7B on the Counseling Bench.
• It approaches the performance of ChatGPT in automated evaluations.
• Fine-tuning on counseling-specific data yields improvements beyond base models and generic instruction-tuned variants.
• A relatively small, high-quality dataset (Psych8k) can significantly enhance domain-specific counseling performance.
• GPT-4-based automated evaluation complements manual assessments and supports rapid benchmarking.
• The approach demonstrates the value of real-world professional data for specialized dialogue systems.