[Paper Review] An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild

Panagiotis Antoniadis, Ioannis Pikoulis+2 more|arXiv (Cornell University)|Jul 7, 2021

Emotion and Mood RecognitionPsychology43 references30 citations

TL;DR

This paper proposes a multi-stream, multi-modal audiovisual emotion recognition framework that integrates facial, bodily, and contextual visual features with audio modality using a CNN-RNN backbone. It achieves state-of-the-art performance on the Aff-Wild2 validation set by demonstrating that body and scene context significantly improve emotion recognition, especially under challenging in-the-wild conditions.

ABSTRACT

In this work we tackle the task of video-based audio-visual emotion recognition, within the premises of the 2nd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW2). Poor illumination conditions, head/body orientation and low image resolution constitute factors that can potentially hinder performance in case of methodologies that solely rely on the extraction and analysis of facial features. In order to alleviate this problem, we leverage both bodily and contextual features, as part of a broader emotion recognition framework. We choose to use a standard CNN-RNN cascade as the backbone of our proposed model for sequence-to-sequence (seq2seq) learning. Apart from learning through the RGB input modality, we construct an aural stream which operates on sequences of extracted mel-spectrograms. Our extensive experiments on the challenging and newly assembled Aff-Wild2 dataset verify the validity of our intuitive multi-stream and multi-modal approach towards emotion recognition in-the-wild. Emphasis is being laid on the the beneficial influence of the human body and scene context, as aspects of the emotion recognition process that have been left relatively unexplored up to this point. All the code was implemented using PyTorch and is publicly available.

Motivation & Objective

• Address the limitations of facial-only emotion recognition in real-world (in-the-wild) settings due to poor lighting, occlusions, and low resolution.
• Investigate the impact of incorporating body posture and scene context as supplementary cues for emotion recognition.
• Improve performance on both categorical (expression) and continuous (valence-arousal) emotion recognition tasks using multi-modal fusion.
• Demonstrate the effectiveness of an audio stream based on mel-spectrograms in conjunction with visual streams.
• Provide a publicly available PyTorch implementation to support reproducibility and further research in affective behavior analysis.

Proposed method

• Employ a standard CNN-RNN (ConvNet + LSTM) cascade for sequence-to-sequence learning on video sequences.
• Use a 50-layer ResNet as the backbone for all visual feature extractors, pre-trained on task-specific datasets.
• Construct three visual streams: (1) face-only (F), (2) body and context features (BCF), and (3) fused face + body + context.
• Implement an aural stream processing sequences of mel-spectrograms extracted from audio, using the same CNN-RNN architecture.
• Fuse audio and visual streams via late late-fusion of feature representations and weighted average of model outputs.
• Apply a visual-semantic embedding loss (Lemb) using GloVe embeddings to align visual and textual representations, with λemb = 1.0.

Experimental results

Research questions

• RQ1 To what extent do body and scene context improve emotion recognition performance in unconstrained, in-the-wild video settings?
• RQ2 How does the inclusion of audio modality (via mel-spectrograms) affect performance in both categorical and dimensional emotion recognition?
• RQ3 Can a multi-stream, multi-modal architecture outperform single-stream facial-only models under challenging visual conditions?
• RQ4 Does the integration of visual-semantic embedding loss enhance the model's ability to generalize across emotion categories and dimensional representations?
• RQ5 How does the performance of a holistic, multi-stream model compare to top-performing facial-only models on the Aff-Wild2 benchmark?

Key findings

• The fusion of facial, bodily, and contextual visual features (Visual BCF) increased the F1 score from 0.453 (face-only) to 0.517 on the validation set for the expression sub-challenge.
• Adding the audio stream to the best visual stream (BCF) further improved the total score to 0.575 for expression recognition and 0.484 for valence-arousal on the validation set.
• The full three-stream model (audio + face + body + context) achieved the highest performance with a total score of 0.592 on the expression sub-challenge and 0.502 on the valence-arousal sub-challenge.
• On the test set, the model achieved a total score of 0.437 on the expression sub-challenge, a 10% improvement in CCC-A (0.616) compared to the baseline, despite not matching the top submissions.
• The inclusion of body and context features led to a performance boost of 0.018 in F1 score (from 0.312 to 0.329) and 0.029 in accuracy (from 0.593 to 0.611), indicating their value under suboptimal facial conditions.
• The visual-semantic embedding loss (Lemb) contributed to a 0.009 improvement in F1 score when added to the BCF stream, showing its effectiveness in aligning visual and semantic representations.