[Paper Review] Developing Social Robots with Empathetic Non-Verbal Cues Using Large Language Models

Yoon Kyung Lee, Yoonwon Jung+2 more|arXiv (Cornell University)|Aug 31, 2023

Social Robot Interaction and HRI16 citations

TL;DR

The paper designs four empathetic non-verbal cues (Speech, Action, Facial expression, Emotion) labeled SAFE and uses an LLM-based system to generate and align these cues with human counselor-defined social cues for more contextual and authentic social robot interactions.

ABSTRACT

We propose augmenting the empathetic capacities of social robots by integrating non-verbal cues. Our primary contribution is the design and labeling of four types of empathetic non-verbal cues, abbreviated as SAFE: Speech, Action (gesture), Facial expression, and Emotion, in a social robot. These cues are generated using a Large Language Model (LLM). We developed an LLM-based conversational system for the robot and assessed its alignment with social cues as defined by human counselors. Preliminary results show distinct patterns in the robot's responses, such as a preference for calm and positive social emotions like 'joy' and 'lively', and frequent nodding gestures. Despite these tendencies, our approach has led to the development of a social robot capable of context-aware and more authentic interactions. Our work lays the groundwork for future studies on human-robot interactions, emphasizing the essential role of both verbal and non-verbal cues in creating social and empathetic robots.

Motivation & Objective

• Motivate the enhancement of social robots through empathetic non-verbal cues.
• Define and label four cue types (Speech, Action, Facial expression, Emotion) to guide robot behavior.
• Develop an LLM-based conversational system for a robot that aligns with human counselor cues.
• Evaluate the alignment and effectiveness of the generated cues in social interactions.
• Lay groundwork for future human-robot interaction studies emphasizing verbal and non-verbal empathy.

Proposed method

• Propose four empathetic non-verbal cues (SAFE) for social robots: Speech, Action, Facial expression, and Emotion.
• Develop an LLM-based conversational system to generate and control these cues.
• Assess cue alignment by comparing generated cues with standards defined by human counselors.
• Analyze robot response patterns to identify tendencies such as calm/positive emotions and nodding gestures.
• Demonstrate context-aware and more authentic interactions through integrated verbal and non-verbal cues.

Experimental results

Research questions

• RQ1 How can LLM-generated non-verbal cues be aligned with human counselor-defined social cues for empathetic interaction?
• RQ2 What are the observable patterns in robot responses (emotions, gestures) when using SAFE cues?
• RQ3 Can the integrated SAFE framework enable more context-aware and authentic social robot interactions?
• RQ4 What quality and alignment metrics indicate successful empathetic non-verbal cue generation in social robots?

Key findings

• Preliminary results show the robot's responses exhibit a preference for calm and positive social emotions such as 'joy' and 'lively'.
• Frequent nodding gestures were observed in the robot's non-verbal behavior.
• The approach enables context-aware and more authentic interactions between humans and the social robot.
• The SAFE cues, generated from an LLM, provide a structured way to integrate verbal and non-verbal empathy in robotics.
• The work establishes a foundation for future studies on human-robot interaction emphasizing empathy through both verbal and non-verbal channels.