[Paper Review] Real-time Automatic Emotion Recognition from Body Gestures

Stefano Piana, Alessandra Staglianò+3 more|arXiv (Cornell University)|Feb 20, 2014

Human Pose and Action Recognition22 references56 citations

TL;DR

This paper presents a real-time system for automatic emotion recognition from full-body gestures using 3D motion data. It extracts postural, kinematic, and geometrical features from skeletal sequences and classifies emotions using a multi-class SVM, achieving 61.3% accuracy on a six-emotion task—very close to human performance (61.9%)—and validating the approach on both professional motion capture and low-cost Kinect sensors.

ABSTRACT

Although psychological research indicates that bodily expressions convey important affective information, to date research in emotion recognition focused mainly on facial expression or voice analysis. In this paper we propose an approach to realtime automatic emotion recognition from body movements. A set of postural, kinematic, and geometrical features are extracted from sequences 3D skeletons and fed to a multi-class SVM classifier. The proposed method has been assessed on data acquired through two different systems: a professionalgrade optical motion capture system, and Microsoft Kinect. The system has been assessed on a "six emotions" recognition problem, and using a leave-one-subject-out cross validation strategy, reached an overall recognition rate of 61.3% which is very close to the recognition rate of 61.9% obtained by human observers. To provide further testing of the system, two games were developed, where one or two users have to interact to understand and express emotions with their body.

Motivation & Objective

• To develop a real-time system for automatic emotion recognition from body movements, addressing the underexplored role of body language in affective computing.
• To bridge the gap between psychological theories of emotion expression and machine learning by extracting behaviorally meaningful features from 3D motion data.
• To evaluate the system's performance on both high-precision motion capture and low-cost RGB-D sensors like Microsoft Kinect.
• To integrate the emotion recognition module into serious games for children with Autism Spectrum Conditions to improve emotional expressiveness and recognition.
• To enable real-time feedback in interactive applications using feature extraction and classification pipelines.

Proposed method

• The system extracts 3D skeletal data from motion capture or RGB-D cameras (e.g., Kinect), representing full-body movements in real time.
• A set of psychology-inspired features—postural, kinematic, and geometrical—are computed over time, including trunk movement, arm motion, velocity, force, and direction.
• Features are temporally integrated into a feature vector per gesture or movement segment to represent emotional content.
• A multi-class Support Vector Machine (SVM) classifier is trained to recognize six basic emotions: happiness, anger, sadness, fear, disgust, and surprise.
• The system is implemented using the EyesWeb XMI platform for real-time feature extraction and feedback generation.
• Performance is evaluated using a leave-one-subject-out cross-validation strategy on data from both professional and consumer-grade sensors.

Experimental results

Research questions

• RQ1 Can body gestures alone convey sufficient information for accurate real-time automatic emotion recognition?
• RQ2 How does the performance of an automated system based on body motion compare to human recognition accuracy on the same task?
• RQ3 To what extent can low-cost RGB-D sensors like Microsoft Kinect support reliable emotion recognition compared to professional motion capture systems?
• RQ4 Can a feature set derived from psychological models of emotion expression effectively capture emotional content in body movements?
• RQ5 How can such a system be integrated into real-time interactive applications, such as serious games for autism intervention?

Key findings

• The proposed system achieved a recognition accuracy of 61.3% on a six-emotion recognition task using leave-one-subject-out cross-validation.
• This performance is very close to the human observer recognition rate of 61.9%, indicating strong potential for real-world applicability.
• The system demonstrated robustness across two data sources: professional optical motion capture and Microsoft Kinect, validating its feasibility with low-cost hardware.
• The integration of the emotion recognition module into two serious games—'Body Emotion Game' and 'Emotional Charades'—demonstrated its viability for real-time, interactive feedback in therapeutic contexts.
• The system successfully enabled real-time emotion inference from body movements, supporting interactive learning for children with Autism Spectrum Conditions.
• The results suggest that body movement features derived from psychological theories can effectively represent emotional states in machine learning models.