[Paper Review] CBT-LLM: A Chinese Large Language Model for Cognitive Behavioral Therapy-based Mental Health Question Answering

Hongbin Na|arXiv (Cornell University)|Mar 24, 2024

Mental Health via Writing6 citations

TL;DR

CBT-LLM fine-tunes a Chinese LLM with a CBT-focused dataset generated via prompts, achieving structured, CBT-aligned mental health Q&A performance over baselines.

ABSTRACT

The recent advancements in artificial intelligence highlight the potential of language models in psychological health support. While models trained on data from mental health service platform have achieved preliminary success, challenges persist in areas such as data scarcity, quality, and ensuring a solid foundation in psychological techniques. To address these challenges, this study introduces a novel approach to enhance the precision and efficacy of psychological support through large language models. Specifically, we design a specific prompt derived from principles of Cognitive Behavioral Therapy (CBT) and have generated the CBT QA dataset, specifically for Chinese psychological health Q&A based on CBT structured intervention strategies. Unlike previous methods, our dataset emphasizes professional and structured response. Utilizing this dataset, we fine-tuned the large language model, giving birth to CBT-LLM, the large-scale language model specifically designed for Cognitive Behavioral Therapy techniques. Empirical evaluations demonstrate that CBT-LLM excels in generating structured, professional, and highly relevant responses in psychological health support tasks, showcasing its practicality and quality. The model is available on Hugging Face: https://huggingface.co/Hongbin37/CBT-LLM.

Motivation & Objective

• Design a CBT-specific prompt and CBT QA dataset for Chinese mental health dialogues.
• Fine-tune a large language model with instruction tuning and LoRA for CBT Q&A tasks.
• Evaluate the model against baselines using automatic metrics and human judgments.

Proposed method

• Create CBT-oriented responses for PsyQA questions using a CBT prompt and ChatGPT to form the CBT QA dataset.
• Fine-tune a Transformer-Decoder LLM with instruction tuning and Low-Rank Adaptation (LoRA) using the CBT QA dataset.
• Train with cross-entropy loss on a 3-epoch schedule, using 16-bit precision and a cosine learning-rate schedule.

Experimental results

Research questions

• RQ1 Can a CBT-focused prompt and dataset improve the CBT-structure, relevance, and usefulness of Chinese mental health Q&A produced by LLMs?
• RQ2 How does CBT-LLM compare to Chinese-capable baselines on automatic and human evaluations for CBT-oriented responses?
• RQ3 What is the quality of CBT distortion recognition and how reliable is the CBT prompt in generating CBT-consistent outputs?

Key findings

• CBT-LLM yields higher automatic metric scores than baselines on the CBT QA task (Baichuan-7B, Alpaca-Chinese-7B, LLaMA-Chinese-7B, Qwen-7B).
• Human evaluation shows Baichuan-7B achieving the best relevance, structure adherence, and helpfulness among baselines in CBT contexts.
• The CBT prompt can identify cognitive distortions with 0.69 accuracy, 0.93 recall, and 0.65 F1 on a 500-sample psychotherapist-annotated subset.
• CBT-LLM outputs are described as structured, professional, and highly relevant to user needs in CBT-based psychological support.