[Paper Review] Building Emotional Support Chatbots in the Era of LLMs

Zhonghua Zheng, Lizi Liao+2 more|arXiv (Cornell University)|Aug 17, 2023

Mental Health via Writing12 citations

TL;DR

The paper builds an extensible emotional support dialogue dataset (ExTES) by combining human seed dialogues with in-context generation from LLMs, then fine-tunes LLaMA-based models with parameter-efficient methods to create an emotional support chatbot and evaluates its quality, safety, and generalizability.

ABSTRACT

The integration of emotional support into various conversational scenarios presents profound societal benefits, such as social interactions, mental health counseling, and customer service. However, there are unsolved challenges that hinder real-world applications in this field, including limited data availability and the absence of well-accepted model training paradigms. This work endeavors to navigate these challenges by harnessing the capabilities of Large Language Models (LLMs). We introduce an innovative methodology that synthesizes human insights with the computational prowess of LLMs to curate an extensive emotional support dialogue dataset. Our approach is initiated with a meticulously designed set of dialogues spanning diverse scenarios as generative seeds. By utilizing the in-context learning potential of ChatGPT, we recursively generate an ExTensible Emotional Support dialogue dataset, named ExTES. Following this, we deploy advanced tuning techniques on the LLaMA model, examining the impact of diverse training strategies, ultimately yielding an LLM meticulously optimized for emotional support interactions. An exhaustive assessment of the resultant model showcases its proficiency in offering emotional support, marking a pivotal step in the realm of emotional support bots and paving the way for subsequent research and implementations.

Motivation & Objective

• Address data scarcity and training challenges in emotional support (ES) conversations by leveraging LLMs.
• Create a large, diverse ES dialogue corpus (ExTES) with scenarios and strategies.
• Evaluate multiple parameter-efficient fine-tuning techniques for ES chatbots on LLaMA.
• Assess dataset toxicity and cross-dataset generalization to ESConv and ExTES.
• Provide insights into strategy distributions and conversation dynamics in ES dialogues.

Proposed method

• Construct 36 ES scenarios and 16 ES strategies to seed dialogues.
• Manually curate 87 seed dialogues from existing ES datasets and web sources.
• Use ChatGPT in self-chat loop to generate ~11k ES dialogues (ExTES) guided by seeds.
• Manually review and correct generated dialogues to ensure quality.
• Fine-tune LLaMA-7B family models with parameter-efficient methods (LoRA, Adapter) and compare to DialoGPT baseline.
• Evaluate with automatic metrics (PPL, METEOR, BLEU/D, ROUGE, Extrema, D-1/2/3) and human evaluation; assess toxicity with Perspective API; perform cross-dataset testing (ExTES vs ESConv).

Experimental results

Research questions

• RQ1 How effective are various parameter-efficient fine-tuning strategies (LoRA, Adapter) on LLaMA for emotional support dialogue?
• RQ2 Does ExTES generalize across ES domains and outperform ESConv on ES tasks?
• RQ3 What is the toxicity level of ExTES-generated data and can it be further reduced with fine-tuning?
• RQ4 How does dataset size and strategy-guided generation affect quality and user-perceived support?

Key findings

• LoRA-tuning on LLaMA generally outperforms other backbones (DialoGPT, Adapter) across automatic metrics for ES dialogue.
• Strategy-guided variants improve some aspects (e.g., targeted suggestions) but may reduce diversity in others.
• ExTES achieves comparable or better human-evaluated quality (informativeness, helpfulness) than seed dialogues and ESConv in several metrics.
• Cross-dataset experiments show models trained on ExTES perform well on ESConv test sets and better generalize to unseen ES scenarios than ESConv-trained models.
• Toxicity scores on ExTES are low and further reduced with LoRA-tuning, indicating safer ES interactions.
• ExTES demonstrates strong generality for ES applications beyond the original ES context, enabling robust ES chatbot development.