[Paper Review] Body Gesture Recognition to Control a Social Robot

Javier Laplaza, Joan Jaume Oliver+3 more|arXiv (Cornell University)|Jun 15, 2022

Robotics and Automated Systems4 citations

TL;DR

This paper proposes a neural network-based body gesture recognition system using a custom dataset of natural human gestures to enable intuitive, non-verbal communication with social robots. The model achieves over 80% accuracy in classifying static full-body gestures, and experiments show users perceive gesture-recognizing robots as more natural, sociable, and comfortable than those relying on remote control.

ABSTRACT

In this work, we propose a gesture based language to allow humans to interact with robots using their body in a natural way. We have created a new gesture detection model using neural networks and a custom dataset of humans performing a set of body gestures to train our network. Furthermore, we compare body gesture communication with other communication channels to acknowledge the importance of adding this knowledge to robots. The presented approach is extensively validated in diverse simulations and real-life experiments with non-trained volunteers. This attains remarkable results and shows that it is a valuable framework for social robotics applications, such as human robot collaboration or human-robot interaction.

Motivation & Objective

• To develop a natural, gesture-based communication method for human-robot interaction without requiring prior robot experience.
• To create a custom dataset of full-body gestures performed by non-expert volunteers.
• To evaluate gesture-based interaction against traditional remote control in terms of user perception and engagement.
• To investigate whether gesture recognition enhances perceived sociability, naturalness, and comfort in human-robot interaction.

Proposed method

• A custom dataset of 15 full-body gestures was collected from 15 non-trained volunteers performing static poses.
• MediaPipe Pose was used to extract 33 3D body landmarks (x, y, z, visibility) from the final frame of video sequences.
• A four-layer fully connected neural network with ReLU activations was trained to classify gestures based on the 33-body landmark coordinates.
• The model uses a linear classifier with batch normalization and dropout for regularization to improve generalization.
• Experiments compared gesture recognition with remote control using the IVO robot in real-world settings with non-expert users.
• User perception was evaluated via surveys measuring sociability, naturalness, security, and comfort using a 7-point Likert scale.

Experimental results

Research questions

• RQ1 How do non-expert users rate gesture-based communication versus remote control in human-robot interaction?
• RQ2 To what extent does gesture recognition improve perceived naturalness and sociability of a social robot?
• RQ3 Can a deep learning model accurately classify static full-body gestures using only the final frame of video input?
• RQ4 How does gesture recognition affect user comfort and sense of control during robot interaction?
• RQ5 What is the impact of gesture recognition on the perceived safety and engagement in human-robot collaboration?

Key findings

• The gesture recognition model achieved over 80% classification accuracy for most gestures, except for visually similar ones.
• Users rated gesture-based interaction significantly higher in naturalness, sociability, comfort, and security compared to remote control (p < 0.05).
• Participants perceived the robot as more human-like and social when it understood their gestures, leading to longer interactions.
• The robot’s use of speech and verbal feedback was perceived as appropriate and enhanced the interaction experience.
• The study confirms that gesture recognition significantly improves user perception and acceptance of social robots in real-world settings.