[Paper Review] Vision based body gesture meta features for Affective Computing

Indigo Orton|arXiv (Cornell University)|Feb 10, 2020

Emotion and Mood Recognition49 references4 citations

TL;DR

This paper introduces novel vision-based body gesture meta-features—capturing movement, duration, and surprise—derived from pose estimation to predict psychological distress, particularly depression. Using a new dataset of 65 interview videos with self-reported distress labels, the method achieves an 82.70% F1 score in depression prediction via linear regression, demonstrating that aggregated gesture meta-features offer a promising, interpretable alternative to traditional hand-crafted or deep learning-based approaches in affective computing.

ABSTRACT

Early detection of psychological distress is key to effective treatment. Automatic detection of distress, such as depression, is an active area of research. Current approaches utilise vocal, facial, and bodily modalities. Of these, the bodily modality is the least investigated, partially due to the difficulty in extracting bodily representations from videos, and partially due to the lack of viable datasets. Existing body modality approaches use automatic categorization of expressions to represent body language as a series of specific expressions, much like words within natural language. In this dissertation I present a new type of feature, within the body modality, that represents meta information of gestures, such as speed, and use it to predict a non-clinical depression label. This differs to existing work by representing overall behaviour as a small set of aggregated meta features derived from a person's movement. In my method I extract pose estimation from videos, detect gestures within body parts, extract meta information from individual gestures, and finally aggregate these features to generate a small feature vector for use in prediction tasks. I introduce a new dataset of 65 video recordings of interviews with self-evaluated distress, personality, and demographic labels. This dataset enables the development of features utilising the whole body in distress detection tasks. I evaluate my newly introduced meta-features for predicting depression, anxiety, perceived stress, somatic stress, five standard personality measures, and gender. A linear regression based classifier using these features achieves a 82.70% F1 score for predicting depression within my novel dataset.

Motivation & Objective

• To address the underexplored body modality in automatic distress detection by introducing meta-features that capture behavioral patterns beyond discrete gestures.
• To overcome the lack of publicly available datasets with source videos and pose data by creating a new dataset of 65 interview recordings with self-evaluated distress, personality, and demographic labels.
• To develop a feature extraction pipeline that transforms raw pose sequences into interpretable, low-dimensional meta-features representing movement dynamics, duration, and surprise.
• To validate the predictive power of these meta-features for non-clinical depression, anxiety, stress, personality traits, and gender using traditional and machine learning classifiers.

Proposed method

• Pose estimation is extracted from video recordings using open-source tools to obtain 2D joint coordinates of body parts.
• Gestures are detected as sequences of movement in specific body parts (e.g., arms, hands) based on changes in joint positions over time.
• Meta-features are computed per gesture, including movement magnitude (distance traveled), duration (time span), and surprise (deviation from expected gesture timing based on prior patterns).
• These per-gesture features are aggregated across all gestures in a video using statistical summaries—mean and standard deviation—resulting in a compact, low-dimensional feature vector.
• A linear regression classifier is trained on the aggregated meta-features to predict depression and other psychological states.
• The method is evaluated on a novel dataset of 65 interview videos with self-reported distress and demographic labels, enabling full pipeline validation.

Experimental results

Research questions

• RQ1 Can meta-features derived from body gesture dynamics—such as movement, duration, and surprise—predict psychological distress more effectively than discrete gesture categorization?
• RQ2 To what extent do these meta-features generalize across multiple psychological states, including depression, anxiety, perceived stress, and personality traits?
• RQ3 How does the performance of a linear classifier using meta-features compare to existing approaches in the absence of large-scale public datasets?
• RQ4 Can the proposed meta-features be interpreted in light of psychological literature on distress behaviors, such as fidgeting or rhythmic movements?
• RQ5 What is the potential of these features in multi-modal or deep learning frameworks, particularly when combined with other modalities like facial expressions or speech?

Key findings

• The proposed gesture meta-features achieved an 82.70% F1 score in classifying depression using a linear regression classifier on the novel dataset.
• The method outperformed baseline approaches that rely on discrete gesture categorization, demonstrating the value of aggregated dynamic features over isolated gesture labels.
• The meta-features showed strong predictive power not only for depression but also for anxiety, perceived stress, somatic stress, and five standard personality traits.
• The surprise feature, measuring deviation from expected gesture timing, showed potential for capturing unexpected or disrupted behaviors linked to distress.
• The approach is interpretable and grounded in psychological theory, as it models behaviors such as fidgeting and rhythmic movement through statistical measures of timing and repetition.
• The study highlights the feasibility of using pose estimation and simple statistical aggregation to create effective, lightweight features for affective computing, even with limited data.