[Paper Review] Affect Intensity Estimation Using Multiple Modalities

Amol Patwardhan, Gerald M. Knapp|arXiv (Cornell University)|Jul 5, 2016

Emotion and Mood Recognition3 citations

TL;DR

This paper proposes a multimodal affect intensity estimation model that fuses facial, body, hand, and speech cues using a weighted sum of classification confidence, feature point displacement, and motion speed. Results show speech and hand modalities significantly improve intensity estimation accuracy on a 0–1 arousal-based scale, outperforming unimodal approaches.

ABSTRACT

One of the challenges in affect recognition is accurate estimation of the emotion intensity level. This research proposes development of an affect intensity estimation model based on a weighted sum of classification confidence levels, displacement of feature points and speed of feature point motion. The parameters of the model were calculated from data captured using multiple modalities such as face, body posture, hand movement and speech. A preliminary study was conducted to compare the accuracy of the model with the annotated intensity levels. An emotion intensity scale ranging from 0 to 1 along the arousal dimension in the emotion space was used. Results indicated speech and hand modality significantly contributed in improving accuracy in emotion intensity estimation using the proposed model.

Motivation & Objective

• To address the challenge of accurately estimating emotion intensity in human-computer interaction.
• To develop a model that integrates multiple modalities—face, body, hand, and speech—for improved intensity estimation.
• To evaluate the contribution of individual modalities to the overall accuracy of affect intensity prediction.
• To calibrate model parameters using data captured via Kinect and audio sensors.
• To validate the model against annotated intensity levels on a continuous 0–1 arousal scale.

Proposed method

• The model computes affect intensity as a weighted sum of classification confidence levels from each modality.
• Feature point displacement and motion speed are extracted from facial and body movements using Kinect-based tracking.
• Speech features are processed to estimate emotional intensity, contributing to the final weighted score.
• Model parameters are optimized using training data from multiple modalities, including face, body posture, hand motion, and speech.
• The final intensity estimate is derived from a fusion of confidence, displacement, and motion speed across modalities.
• A 0–1 scale along the arousal dimension is used to represent continuous emotion intensity levels.

Experimental results

Research questions

• RQ1 How does combining multiple modalities improve affect intensity estimation compared to unimodal approaches?
• RQ2 Which modalities—face, body, hand, or speech—contribute most significantly to intensity estimation accuracy?
• RQ3 Can feature displacement and motion speed enhance the estimation of emotion intensity in real-time systems?
• RQ4 How do classification confidence levels from different modalities interact in a weighted fusion model?
• RQ5 To what extent does the proposed model align with human-annotated emotion intensity levels?

Key findings

• Speech and hand modalities significantly improved the accuracy of affect intensity estimation in the proposed model.
• The weighted sum fusion of confidence, displacement, and motion speed yielded better performance than individual modalities.
• The model achieved improved alignment with human-annotated intensity levels on a 0–1 arousal scale.
• Face and body modalities contributed to the model but were less impactful than speech and hand.
• The use of Kinect-enabled motion tracking enhanced feature extraction for posture and gesture analysis.
• The model demonstrated feasibility for real-time affect intensity estimation in multimodal human-computer interaction.