[Paper Review] SoulChat: Improving LLMs' Empathy, Listening, and Comfort Abilities through Fine-tuning with Multi-turn Empathy Conversations

Yirong Chen, Xiaofen Xing+5 more|arXiv (Cornell University)|Nov 1, 2023

Topic Modeling4 citations

TL;DR

This paper introduces SoulChat, a large-scale Chinese multi-turn empathy conversation dataset with over 2.3 million samples, constructed by fine-tuning LLMs using human-centered prompts to enhance empathy, listening, and comfort abilities. Fine-tuning on SoulChatCorpus significantly improves LLMs' performance in emotional support scenarios, outperforming baseline models in both automatic and human evaluations.

ABSTRACT

Large language models (LLMs) have been widely applied in various fields due to their excellent capability for memorizing knowledge and chain of thought (CoT). When these language models are applied in the field of psychological counseling, they often rush to provide universal advice. However, when users seek psychological support, they need to gain empathy, trust, understanding and comfort, rather than just reasonable advice. To this end, we constructed a multi-turn empathetic conversation dataset of more than 2 million samples, in which the input is the multi-turn conversation context, and the target is empathetic responses that cover expressions such as questioning, comfort, recognition, listening, trust, emotional support, etc. Experiments have shown that the empathy ability of LLMs can be significantly enhanced when finetuning by using multi-turn dialogue history and responses that are closer to the expression of a psychological consultant.

Motivation & Objective

• To address the lack of large-scale, multi-turn, Chinese empathy-focused dialogue datasets for fine-tuning LLMs in mental health applications.
• To improve LLMs' ability to provide empathetic responses—such as listening, comfort, recognition, and emotional support—rather than generic advice.
• To develop a human-centered LLM that better emulates psychological counselors by leveraging multi-turn dialogue patterns and empathy constraints.
• To create a scalable, privacy-protected dataset using data outsourcing, rule-based cleaning, and human-annotated refinement.
• To evaluate the effectiveness of fine-tuning on empathy-specific responses using both automatic metrics and expert-annotated human evaluation.

Proposed method

• Constructed a 2.3M-sample multi-turn empathy conversation dataset, SoulChatCorpus, by first collecting 619,725 single-turn psychological counseling responses and 215,813 questions across 12 topics.
• Applied a strengthened Chinese empathy constraint prompt to fine-tune GPT-based models, emphasizing expressions like listening, comfort, trust, recognition, and emotional support.
• Used ChatGPT to convert single-turn dialogues into multi-turn empathetic conversations, followed by rule-based filtering and manual proofreading to enhance empathy expression.
• Conducted human evaluation with three psychology experts rating responses on naturalness, empathy, helpfulness, and safety using a 3-point scale (0–2) and perfect agreement (κ=1) on safety.
• Fine-tuned the base model ChatGLM-6B on SoulChatCorpus and evaluated performance using BLEU, ROUGE, and human annotations on both SoulChatCorpus and SMILECHAT test sets.
• Employed a rigorous data curation pipeline, including removal of sensitive content (e.g., 'suicide', 'I am') and 105,134 harmful samples, to ensure ethical compliance.

Experimental results

Research questions

• RQ1 Can fine-tuning LLMs on a large-scale, multi-turn, Chinese empathy conversation dataset significantly improve their ability to provide empathetic, listening-oriented, and comforting responses?
• RQ2 How does the proposed empathy constraint prompt compare to existing prompts (e.g., SMILE) in enhancing expression of empathy in LLM-generated responses?
• RQ3 To what extent does the SoulChatCorpus dataset improve zero-shot performance on other empathy benchmarks like SMILECHAT?
• RQ4 How do human-annotated evaluations compare to automatic metrics in assessing empathy and safety in LLM-generated counseling responses?
• RQ5 What are the key limitations in generating user-personalized empathy responses, and how can future models account for user attributes like age, gender, and personality?

Key findings

• SoulChat outperformed ChatGLM-6B, ChatGPT, and MeChat across all automatic evaluation metrics (BLEU-1 to BLEU-4, ROUGE-1 to ROUGE-L) on both SoulChatCorpus and SMILECHAT test sets.
• In human evaluation, SoulChat achieved a mean empathy score of 1.78 on a 0–2 scale, significantly higher than the baseline models, indicating stronger empathetic response generation.
• The model demonstrated moderate to substantial inter-annotator agreement (Fleiss’ κ = 0.489–0.532) for empathy, helpfulness, and naturalness, with perfect agreement (κ=1) on safety.
• The dataset achieved a 2.3 million-sample scale, making it the first million-scale multi-turn empathy conversation dataset in Chinese mental health and emotional support.
• SoulChat showed strong zero-shot generalization on the SMILECHAT benchmark, indicating transferability of the fine-tuned representations to other empathy-focused tasks.
• The safety assessment confirmed that all potentially harmful content was removed during data curation, and the final model maintained high safety scores in human evaluation.