[Paper Review] Multiple Emotion Descriptors Estimation at the ABAW3 Challenge

Didan Deng|arXiv (Cornell University)|Mar 24, 2022

Emotion and Mood Recognition4 citations

TL;DR

This paper proposes SMM-EmotionNet, a novel multi-task deep learning architecture that separately models facial action units (sign vehicles) and emotional messages (valence, arousal, discrete expressions) based on psychological theory. By leveraging distinct branches for sign vehicles and messages, and applying temporal smoothing, the model achieves state-of-the-art performance on the ABAW3 challenge, outperforming both single-task and multi-task baselines across all metrics.

ABSTRACT

To describe complex emotional states, psychologists have proposed multiple emotion descriptors: sparse descriptors like facial action units; continuous descriptors like valence and arousal; and discrete class descriptors like happiness and anger. According to Ekman and Friesen, 1969, facial action units are sign vehicles that convey the emotion message, while discrete or continuous emotion descriptors are the messages perceived and expressed by human. In this paper, we designed an architecture for multiple emotion descriptors estimation in participating the ABAW3 Challenge. Based on the theory of Ekman and Friesen, 1969, we designed distinct architectures to measure the sign vehicles (i.e., facial action units) and the message (i.e., discrete emotions, valence and arousal) given their different properties. The quantitative experiments on the ABAW3 challenge dataset has shown the superior performance of our approach over two baseline models.

Motivation & Objective

• To address the challenge of estimating multiple emotion descriptors—facial action units (AUs), valence/arousal (VA), and discrete emotions—simultaneously in video sequences.
• To incorporate psychological theory (Ekman & Friesen, 1969) into model architecture design, distinguishing between sign vehicles (AUs) and emotional messages (EXPR, VA) due to their differing observer dependency and behavioral properties.
• To improve performance in low-annotation regimes by using psychological priors to regularize feature learning and reduce inter-observer variability in emotion estimation.
• To explore temporal modeling of emotional features through simple smoothing to reduce noise and improve consistency in sequential predictions.

Proposed method

• Design a dual-branch architecture: one branch for sign vehicles (AUs) using intra- and inter-AU attention mechanisms adapted from Jacob & Stenger (2021), and another for emotional messages (EXPR, VA) via region-based feature projection to a shared message space.
• Project visual features from different facial regions into a shared message space and average them to achieve consensus on emotional messages, reducing observer-dependent variability.
• Apply temporal smoothing using a recursive filter: $\Lambda_t = \frac{1}{1+\mu}(\epsilon_t + \mu\Lambda_{t-1})$, where $\mu$ is tuned via grid search to optimize performance on AU, EXPR, and VA tasks.
• Use a shared feature extractor (e.g., InceptionV3 or VGG-Face) followed by task-specific fully connected heads for AU detection, discrete expression classification, and continuous VA regression.
• Train the model with a cosine annealing learning rate schedule over 300,000 iterations, using cross-entropy for classification and mean squared error for regression.
• Optimize using a multitask loss combining F1 for AU and EXPR, and CCC for VA, with equal weighting in the final evaluation metric.

Experimental results

Research questions

• RQ1 How can psychological distinctions between sign vehicles (e.g., AUs) and emotional messages (e.g., valence, arousal, discrete emotions) be effectively modeled in a deep learning architecture?
• RQ2 Can separating the learning of sign vehicles and emotional messages improve performance on multiple emotion descriptor estimation tasks, especially under sparse annotation?
• RQ3 Does temporal smoothing of features in the message and sign vehicle spaces enhance model robustness and performance on sequential emotion recognition?
• RQ4 To what extent does incorporating psychological priors (e.g., observer-dependent perception, co-occurrence of AUs) improve generalization and reduce inter-observer variance in emotion estimation?

Key findings

• The proposed SMM-EmotionNet achieved a validation F1-AU score of 0.548, outperforming the baseline InceptionV3 model (0.536) and VGG-Face (0.39) on the ABAW3 challenge's AU detection task.
• For discrete emotion classification, the model achieved a macro F1 score of 0.518, surpassing the InceptionV3 baseline (0.489) and VGG-Face (0.23).
• On continuous emotion prediction, the model achieved a CCC of 0.447 for valence and 0.499 for arousal, exceeding the InceptionV3 baseline (0.441 and 0.485, respectively).
• The temporal smoothing approach improved the multitask evaluation metric (Equation 9) from 1.539 (static) to 1.664 on the validation sets, and from 1.104 to 1.113 on the test set, indicating consistent gains across tasks.
• The three-fold cross-validation showed that optimal smoothing hyperparameters were $\mu=7$ for AU and $\mu=9$ for EXPR and VA, with the highest CCC-VA of 0.577 in fold 2.
• The model demonstrated superior generalization and robustness, particularly in handling ambiguous or conflicting facial cues, by leveraging consensus from multiple facial regions in the message space.