[Paper Review] A Multi-term and Multi-task Analyzing Framework for Affective Analysis in-the-wild

Sachihiro Youoku, Yuushi Toyoda+5 more|arXiv (Cornell University)|Sep 29, 2020

Emotion and Mood Recognition20 references4 citations

TL;DR

This paper proposes a multi-term, multi-task framework for affective analysis in-the-wild using video data from the Aff-Wild2 dataset. It extracts multimodal features (action units, head pose, gaze, posture, deep features) across short-, middle-, and long-term time windows, ensembles models per time window, and fuses valence-arousal and expression prediction tasks via multi-task learning, achieving a validation score of 0.498 for valence-arousal and 0.471 for expression.

ABSTRACT

Human affective recognition is an important factor in human-computer interaction. However, the method development with in-the-wild data is not yet accurate enough for practical usage. In this paper, we introduce the affective recognition method focusing on valence-arousal (VA) and expression (EXP) that was submitted to the Affective Behavior Analysis in-the-wild (ABAW) 2020 Contest. Since we considered that affective behaviors have many observable features that have their own time frames, we introduced multiple optimized time windows (short-term, middle-term, and long-term) into our analyzing framework for extracting feature parameters from video data. Moreover, multiple modality data are used, including action units, head poses, gaze, posture, and ResNet 50 or Efficient NET features, and are optimized during the extraction of these features. Then, we generated affective recognition models for each time window and ensembled these models together. Also, we fussed the valence, arousal, and expression models together to enable the multi-task learning, considering the fact that the basic psychological states behind facial expressions are closely related to each another. In the validation set, our model achieved a valence-arousal score of 0.498 and a facial expression score of 0.471. These verification results reveal that our proposed framework can improve estimation accuracy and robustness effectively.

Motivation & Objective

• To improve affective analysis accuracy in real-world (in-the-wild) video data, where variability in lighting, pose, and expression complicates emotion recognition.
• To address the challenge that emotional expressions exhibit different temporal dynamics, requiring multiple time-scale analysis for optimal feature capture.
• To enhance model robustness and performance by leveraging multi-task learning across valence-arousal and facial expression prediction.
• To optimize feature extraction across modalities (e.g., OpenFace, ResNet50, EfficientNet, OpenPose) using dimensionality reduction and importance-based selection.
• To validate the effectiveness of data balancing, time-window expansion (including 3-second windows), and feature selection in boosting model generalization.

Proposed method

• Extracted multimodal features from video frames using pre-trained models: OpenFace (49D: AUs, head pose, gaze), ResNet50/EfficientNet (2048D → 200D/300D via PCA), and OpenPose (75D posture features).
• Applied multiple time windows—short-term (1s), middle-term (6s), long-term (12s)—with 0.2s shift to generate multi-term features capturing dynamic emotional cues.
• Trained separate LightGBM models for valence-arousal (regression with custom metric: 2×CCC − MSE) and expression (multiclass, 7 classes, metric: Eq. 3).
• Ensembled models across time windows using early stopping and hyperparameter tuning (grid search on learning_rate, num_leaves, max_depth, min_child_samples).
• Fused valence-arousal and expression models via multi-task learning to exploit shared representations and improve generalization.
• Optimized model performance through data balancing, inclusion of 3-second window data, and feature selection based on LightGBM feature importances.

Experimental results

Research questions

• RQ1 Can modeling affective states across multiple temporal scales (short-, middle-, long-term) improve the accuracy of valence-arousal and expression estimation in in-the-wild videos?
• RQ2 Does fusing multiple recognition tasks (valence-arousal and expression) lead to better performance than single-task models?
• RQ3 How do different feature extraction strategies (e.g., ResNet50 vs. EfficientNet, PCA reduction, feature selection) impact model robustness and accuracy?
• RQ4 To what extent does data balancing and inclusion of longer time windows (e.g., 3s) improve model generalization on real-world affective video data?
• RQ5 Can multi-term modeling effectively capture the temporal dynamics of emotional expressions such as surprise (short-term) or confusion (longer-term)?

Key findings

• The multi-term model outperformed single-term models, achieving a valence-arousal score of 0.480 on the validation set, up from 0.396 in the middle-term baseline.
• The multi-task model achieved a valence-arousal score of 0.482 and an expression score of 0.432, significantly outperforming the baseline (0.190 and 0.360, respectively).
• Incorporating a 3-second window and feature selection improved performance, with the best submission (submit.4) achieving a valence-arousal score of 0.479 and expression score of 0.495.
• The final model using EfficientNet, data balancing, 3-second window, and feature extraction achieved a valence-arousal score of 0.498 and expression score of 0.471 on the validation set.
• The ablation study confirmed that data balancing and feature selection contributed to improved robustness and generalization across diverse in-the-wild conditions.
• The results demonstrate that multi-task learning and multi-term modeling jointly enhance performance by capturing both temporal dynamics and shared affective representations.