[Paper Review] Data-driven emotional body language generation for social robotics

Mina Marmpena, Fernando García+3 more|arXiv (Cornell University)|May 2, 2022

Social Robot Interaction and HRI4 citations

TL;DR

This paper presents a data-driven framework using a Conditional Variational Autoencoder (CVAE) to generate emotionally expressive robotic body language for social robots, conditioning on valence and arousal levels via latent space sampling. The method successfully generates animations perceived as equally anthropomorphic and animate as hand-designed ones, with high arousal and extreme valence expressions drawing more attention and being rated as more emotional.

ABSTRACT

In social robotics, endowing humanoid robots with the ability to generate bodily expressions of affect can improve human-robot interaction and collaboration, since humans attribute, and perhaps subconsciously anticipate, such traces to perceive an agent as engaging, trustworthy, and socially present. Robotic emotional body language needs to be believable, nuanced and relevant to the context. We implemented a deep learning data-driven framework that learns from a few hand-designed robotic bodily expressions and can generate numerous new ones of similar believability and lifelikeness. The framework uses the Conditional Variational Autoencoder model and a sampling approach based on the geometric properties of the model's latent space to condition the generative process on targeted levels of valence and arousal. The evaluation study found that the anthropomorphism and animacy of the generated expressions are not perceived differently from the hand-designed ones, and the emotional conditioning was adequately differentiable between most levels except the pairs of neutral-positive valence and low-medium arousal. Furthermore, an exploratory analysis of the results reveals a possible impact of the conditioning on the perceived dominance of the robot, as well as on the participants' attention.

Motivation & Objective

• To develop a data-driven method for generating diverse, believable emotional body language (EBL) in humanoid robots.
• To enable precise control over generated EBL based on affective dimensions—valence and arousal—using deep learning.
• To evaluate whether CVAE-generated EBL is perceived as equally anthropomorphic, animate, and emotionally interpretable as hand-designed animations.
• To investigate the impact of emotional conditioning on user attention and perception in human-robot interaction.

Proposed method

• Trained a Conditional Variational Autoencoder (CVAE) on a small dataset of hand-designed EBL animations for the Pepper robot, including motion and eye LED color sequences.
• Conditioned the model on scalar valence labels to control emotional tone (positive, neutral, negative).
• Used geometric properties of the CVAE’s latent space to sample new animations with targeted arousal levels by varying the radius in latent space.
• Applied a sampling strategy that maps desired arousal levels to specific regions in the latent space to generate diverse, contextually relevant animations.
• Validated the model’s generalization capability through user studies assessing perceptual quality and emotional interpretability.
• Employed ordinal logistic regression to analyze user ratings of emotionality, anthropomorphism, and animacy, testing the proportional odds assumption.

Experimental results

Research questions

• RQ1 Can a CVAE model generate emotionally expressive robotic body language that is perceived as believable and lifelike?
• RQ2 To what extent can valence and arousal be effectively controlled in generated EBL animations through conditioning?
• RQ3 How do CVAE-generated animations compare to hand-designed ones in terms of perceived anthropomorphism and animacy?
• RQ4 Do extreme emotional expressions (high arousal or extreme valence) draw more user attention than neutral or low-arousal expressions?
• RQ5 Are there perceptual limitations in differentiating between neutral and positive valence, or between medium and low arousal levels?

Key findings

• Animations conditioned with negative or positive valence were rated significantly higher on emotional interpretability than those with neutral valence (p < 0.001 for negative vs neutral; p = 0.01 for positive vs neutral).
• High arousal animations were perceived as significantly more emotional than medium or low arousal ones (p < 0.001 for high vs medium; p = 0.01 for high vs low).
• No significant difference was found between neutral and positive valence, or between medium and low arousal, suggesting limited perceptual differentiation in these pairs.
• Generated animations were not rated as less anthropomorphic or less animate than hand-designed ones, both pre- and post-study (p > 0.05).
• Animations with high arousal or extreme valence levels drew more user attention, indicating a strong perceptual impact.
• The proportional odds assumption was met for all models (p > 0.05), supporting the validity of the ordinal logistic regression analysis.