[Paper Review] Identifying Emotions from Walking using Affective and Deep Features

Tanmay Randhavane, Uttaran Bhattacharya+4 more|arXiv (Cornell University)|Jun 14, 2019

Human Pose and Action Recognition86 references50 citations

TL;DR

The paper presents a data-driven model that combines LSTM-derived deep gait features with handcrafted affective features from 3D gait data, achieving 80.07% accuracy in classifying perceived emotions (happy, sad, angry, neutral) from walking videos using a Random Forest classifier.

ABSTRACT

We present a new data-driven model and algorithm to identify the perceived emotions of individuals based on their walking styles. Given an RGB video of an individual walking, we extract his/her walking gait in the form of a series of 3D poses. Our goal is to exploit the gait features to classify the emotional state of the human into one of four emotions: happy, sad, angry, or neutral. Our perceived emotion recognition approach uses deep features learned via LSTM on labeled emotion datasets. Furthermore, we combine these features with affective features computed from gaits using posture and movement cues. These features are classified using a Random Forest Classifier. We show that our mapping between the combined feature space and the perceived emotional state provides 80.07% accuracy in identifying the perceived emotions. In addition to classifying discrete categories of emotions, our algorithm also predicts the values of perceived valence and arousal from gaits. We also present an EWalk (Emotion Walk) dataset that consists of videos of walking individuals with gaits and labeled emotions. To the best of our knowledge, this is the first gait-based model to identify perceived emotions from videos of walking individuals.

Motivation & Objective

• Motivate automatic perception of emotions from walking as a non-verbal cue in social perception and HCI contexts.
• Develop a gait-based emotion recognition pipeline that uses both deep temporal features and handcrafted affective features.
• Create and release the EWalk dataset with gait videos and perceived emotion labels for research use.

Proposed method

• Extract 3D gait poses from RGB videos using TimePoseNet to obtain sequences of 16 joints across frames.
• Compute affective features (posture and movement) from gait sequences, including volume, area, distance, angles, stride length, and motion magnitudes (speed, acceleration, jerk).
• Model deep features with an LSTM network trained on multiple gait datasets to capture temporal dynamics of gait.
• Concatenate normalized deep features with affective features and train a Random Forest classifier to predict one of four perceived emotions (happy, angry, sad, neutral).
• Train the LSTM using multiple gait datasets and use a 10-estimator RF (max depth 5) for final classification.
• Evaluate via 10-fold cross-validation on 1384 gaits, reporting accuracy improvements over prior gait-based methods.

Experimental results

Research questions

• RQ1 Can combined deep gait features and affective posture/movement features improve accuracy of perceived emotion recognition from walking compared to using either feature type alone?
• RQ2 What is the best classifier for mapping the fused feature space to discrete perceived emotions in walking?
• RQ3 How well does the approach generalize across acted and non-acted gait datasets?
• RQ4 Can the model predict valence/arousal from gaits in addition to discrete emotion categories?
• RQ5 What is the impact of using a new public gait dataset (EWalk) with perceived emotion labels on performance?

Key findings

• The proposed approach achieves 80.07% accuracy in classifying four perceived emotion categories from gait data.
• Random Forest with normalized concatenated features (deep + affective) outperforms SVM variants for this task.
• The method maintains strong performance on non-acted datasets (CMU and ICT) with 79.72% accuracy.
• A novel dataset, EWalk, is introduced with 1,136 gaits and perceived emotion labels collected via Mechanical Turk.
• The work reports a 13.85% accuracy improvement over other gait-based emotion recognition methods on their evaluation setup.