[Paper Review] MMA-MRNNet: Harnessing Multiple Models of Affect and Dynamic Masked RNN for Precise Facial Expression Intensity Estimation

Dimitrios Kollias, Andreas Psaroudakis+5 more|arXiv (Cornell University)|Mar 1, 2023

Emotion and Mood Recognition9 citations

TL;DR

FaceRNET is a dynamic multi-output Facial Expression Intensity Estimation network that uses per-frame affect representations (VA, AUs, basic expressions) and a Mask-enabled RNN to handle variable video lengths and video-level annotations, achieving state-of-the-art results on Hume-Reaction.

ABSTRACT

This paper presents MMA-MRNNet, a novel deep learning architecture for dynamic multi-output Facial Expression Intensity Estimation (FEIE) from video data. Traditional approaches to this task often rely on complex 3-D CNNs, which require extensive pre-training and assume that facial expressions are uniformly distributed across all frames of a video. These methods struggle to handle videos of varying lengths, often resorting to ad-hoc strategies that either discard valuable information or introduce bias. MMA-MRNNet addresses these challenges through a two-stage process. First, the Multiple Models of Affect (MMA) extractor component is a Multi-Task Learning CNN that concurrently estimates valence-arousal, recognizes basic facial expressions, and detects action units in each frame. These representations are then processed by a Masked RNN component, which captures temporal dependencies and dynamically updates weights according to the true length of the input video, ensuring that only the most relevant features are used for the final prediction. The proposed unimodal non-ensemble learning MMA-MRNNet was evaluated on the Hume-Reaction dataset and demonstrated significantly superior performance, surpassing state-of-the-art methods by a wide margin, regardless of whether they were unimodal, multimodal, or ensemble approaches. Finally, we demonstrated the effectiveness of the MMA component of our proposed method across multiple in-the-wild datasets, where it consistently outperformed all state-of-the-art methods across various metrics.

Motivation & Objective

• Introduce a dynamic multi-output FEIE architecture that works with video-level annotations.
• Leverage per-frame affect representations (VA, AUs, basic expressions) for robust temporal modeling.
• Handle variable input video lengths via a Mask layer with dynamic routing.
• Improve training through a loss that encodes relationships among affect representations.
• Demonstrate state-of-the-art performance on the Hume-Reaction dataset compared to baselines and multimodal methods.

Proposed method

• Affect Representation Extractor Component (REC) is a multi-task CNN that predicts VA, 7 basic expressions, and 17 AUs from each frame.
• REC is trained with a novel loss (L_REC) that includes CCC-based terms, cross-entropy losses, binary cross-entropy, and a distal loss term L_DM that injects prior knowledge about AU–expression relationships.
• An RNN operates on per-frame REC features to model temporal dynamics across the video.
• A Mask layer dynamically routes the RNN outputs according to the true video length, enabling flexible handling of varying frame counts.
• Concatenated routed RNN outputs are fed into a dense layer and then to an output layer predicting seven expression intensities.
• Training uses a Pearson-correlation-based loss across the seven expressions (1 - average ρ) to align with the evaluation metric.

Experimental results

Research questions

• RQ1 Can per-frame affect representations (VA, AUs, basic expressions) extracted by a multi-task REC improve FEIE accuracy on video-level annotations?
• RQ2 Does a Mask-based dynamic routing mechanism effectively handle variable-length videos in a unified FEIE pipeline?
• RQ3 Does incorporating prior knowledge about AU–expression relationships via L_DM improve gradient quality and convergence in REC?
• RQ4 How does FaceRNET compare to state-of-the-art uni- and multi-modal approaches on the Hume-Reaction dataset?
• RQ5 What is the impact of architectural choices (GRU vs LSTM, number of layers/units) and loss variants on FEIE performance?

Key findings

• FaceRNET outperforms state-of-the-art baselines and several multimodal methods on the test set, achieving a Pearson ρ of 0.499.
• Ablation studies show that using all three task representations (VA, basic expressions, AUs) yields the best performance, with higher ρ than using any single task.
• The Mask routing and the proposed L_REC/L_DM losses significantly contribute to performance gains over variants without dynamic routing or with MSE loss.
• The best REC configuration uses a single GRU layer with 128 units followed by a 32-unit dense layer, combined with the Mask layer for variable-length handling.
• REC alone with AU and expression representations provides competitive results, and combined representations maximize performance.