[Paper Review] Empathy Through Multimodality in Conversational Interfaces

Mahyar Abbasian, Iman Azimi+3 more|arXiv (Cornell University)|May 8, 2024

Language, Metaphor, and CognitionPsychology3 citations

TL;DR

This paper proposes a multimodal Conversational Health Agent (CHA) powered by Large Language Models (LLMs) that integrates speech emotion recognition and text-based dialogue to deliver contextually aware, empathetic responses in mental health support. By leveraging the openCHA framework with speech-to-text, emotion detection, and web search, the CHA achieves 89% accuracy in emotion recognition and generates highly empathetic responses, especially for users expressing sadness, as validated by human evaluators scoring average empathy at 7.24/10.

ABSTRACT

Agents represent one of the most emerging applications of Large Language Models (LLMs) and Generative AI, with their effectiveness hinging on multimodal capabilities to navigate complex user environments. Conversational Health Agents (CHAs), a prime example of this, are redefining healthcare by offering nuanced support that transcends textual analysis to incorporate emotional intelligence. This paper introduces an LLM-based CHA engineered for rich, multimodal dialogue-especially in the realm of mental health support. It adeptly interprets and responds to users' emotional states by analyzing multimodal cues, thus delivering contextually aware and empathetically resonant verbal responses. Our implementation leverages the versatile openCHA framework, and our comprehensive evaluation involves neutral prompts expressed in diverse emotional tones: sadness, anger, and joy. We evaluate the consistency and repeatability of the planning capability of the proposed CHA. Furthermore, human evaluators critique the CHA's empathic delivery, with findings revealing a striking concordance between the CHA's outputs and evaluators' assessments. These results affirm the indispensable role of vocal (soon multimodal) emotion recognition in strengthening the empathetic connection built by CHAs, cementing their place at the forefront of interactive, compassionate digital health solutions.

Motivation & Objective

• To develop a Conversational Health Agent (CHA) capable of interpreting emotional cues from speech and text to deliver empathetic, context-aware responses in mental health support.
• To address the limitations of text-only LLM-based agents by integrating multimodal inputs, including vocal emotion and physiological signals, to enhance emotional intelligence in digital interactions.
• To evaluate the consistency and empathetic quality of the CHA’s responses across diverse emotional states—sadness, anger, and joy—using human evaluators.
• To demonstrate that emotion-informed planning improves the relevance and empathy of generated responses in conversational AI.
• To lay the foundation for future CHAs that incorporate facial and physiological modalities to achieve human-like empathetic engagement.

Proposed method

• The CHA is built on the openCHA framework, integrating LLMs with speech-to-text, speech emotion detection, Internet search, and text-to-speech modules.
• Emotional state is detected from voice input using a dedicated speech emotion recognition model, which informs the planning phase.
• The planner uses both the user’s query and detected emotion to generate targeted search queries, improving response personalization.
• Responses are generated by the LLM using retrieved information and emotional context, ensuring empathetic tone and alignment with user sentiment.
• The system maintains emotional context throughout the dialogue by forwarding detected emotion to the response generator, even when search is not used.
• Evaluation involves two stages: assessing planning accuracy and measuring empathetic response quality via human evaluation on a 0–10 scale.

Figure 1: LLM-based CHA for multimodal speech-based emotional support

Experimental results

Research questions

• RQ1 Can an LLM-based CHA accurately detect emotional states from speech and use them to guide response generation?
• RQ2 How does the integration of speech emotion recognition improve the empathy and contextual relevance of conversational responses?
• RQ3 How do responses vary in empathetic quality when users express sadness, anger, or joy?
• RQ4 To what extent does emotion-informed planning enhance the consistency and personalization of CHA responses?
• RQ5 Can multimodal input (speech + text) significantly improve the empathetic capabilities of LLM-based agents in mental health contexts?

Key findings

• The CHA achieved 89% accuracy in detecting emotional states from voice input, demonstrating strong performance in multimodal emotion recognition.
• The planner correctly invoked Internet search tools based on emotional context with 61% accuracy, indicating effective integration of emotion into planning.
• Human evaluators rated responses to questions posed in sadness as the most empathetic, with an average score of 7.24 out of 10.
• Responses to joyful and angry queries received average empathy scores of 6.24 and 6.56, respectively, showing lower but still reasonable alignment.
• The system consistently maintained emotional context across interactions, with evaluators noting that responses were more aligned and empathetic when emotion was properly recognized and utilized.
• The results confirm that integrating speech-based emotion recognition significantly enhances the empathetic quality of LLM-driven conversational agents in mental health applications.

Figure 2: Examples of developed CHA answering a user voice query with Sad (a) and Happy (b) emotions