[Paper Review] Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots

Youngwoo Yoon, Woo-Ri Ko+4 more|arXiv (Cornell University)|Oct 30, 2018

Speech and dialogue systems6 citations

TL;DR

This paper presents an end-to-end deep learning model that generates co-speech gestures for humanoid robots using speech transcripts from 52 hours of TED talks. The model employs an encoder-decoder architecture to produce diverse, contextually relevant gestures—iconic, metaphoric, deictic, and beat—demonstrating human-like motion in real-time on a NAO robot, with subjective evaluations confirming gesture-speech alignment and naturalness.

ABSTRACT

Co-speech gestures enhance interaction experiences between humans as well as between humans and robots. Existing robots use rule-based speech-gesture association, but this requires human labor and prior knowledge of experts to be implemented. We present a learning-based co-speech gesture generation that is learned from 52 h of TED talks. The proposed end-to-end neural network model consists of an encoder for speech text understanding and a decoder to generate a sequence of gestures. The model successfully produces various gestures including iconic, metaphoric, deictic, and beat gestures. In a subjective evaluation, participants reported that the gestures were human-like and matched the speech content. We also demonstrate a co-speech gesture with a NAO robot working in real time.

Motivation & Objective

• To develop a learning-based method for co-speech gesture generation that reduces reliance on expert-designed rules.
• To enable humanoid robots to produce diverse, contextually appropriate gestures during speech using only speech text as input.
• To demonstrate real-time gesture generation on a physical robot (NAO) for enhanced human-robot interaction.
• To evaluate the naturalness and content alignment of generated gestures through subjective human perception studies.

Proposed method

• The model uses a sequence-to-sequence neural network with an encoder to encode speech text into a latent representation.
• A transformer-based decoder generates a sequence of gesture parameters (e.g., joint angles, timing) conditioned on the encoded speech.
• The model is pre-trained on 52 hours of TED talk videos, where speech transcripts are paired with corresponding human gestures.
• Gesture types—iconic, metaphoric, deictic, and beat—are implicitly learned through end-to-end training without explicit labeling.
• The architecture enables end-to-end training from raw text to motion sequences, avoiding handcrafted gesture rules.
• Real-time inference is achieved by deploying the model on a NAO robot, enabling synchronized speech and gesture output.

Experimental results

Research questions

• RQ1 Can a deep learning model generate diverse co-speech gestures from speech text alone, without rule-based systems?
• RQ2 To what extent do the generated gestures match the semantic content of the speech?
• RQ3 How natural and human-like do the generated gestures appear to human observers?
• RQ4 Can the model produce a variety of gesture types (e.g., iconic, beat, deictic) in a single end-to-end framework?
• RQ5 Is real-time gesture generation feasible on a physical humanoid robot using this approach?

Key findings

• The model successfully generates a wide range of gesture types, including iconic, metaphoric, deictic, and beat gestures, without explicit supervision on gesture categories.
• Subjective evaluations showed that participants perceived the generated gestures as natural and well-aligned with speech content.
• The system achieved real-time performance on a NAO humanoid robot, demonstrating practical deployment feasibility.
• The model generalizes across diverse speech content, producing contextually relevant gestures even for unseen utterances.
• The use of TED talk data enabled the model to learn rich, socially meaningful gesture patterns from authentic human speech.