[Paper Review] MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild

Yuanyuan Liu, Wei Dai+5 more|arXiv (Cornell University)|Aug 1, 2022

Emotion and Mood Recognition4 citations

TL;DR

This paper introduces MAFW, a large-scale, in-the-wild, multi-modal database with 10,045 video-audio clips annotated for compound emotions and affective behaviors. It proposes a novel Transformer-based method, T-ESFL, that leverages expression-change relations across modalities to improve dynamic facial expression recognition, outperforming state-of-the-art methods on both uni- and multi-modal FER tasks.

ABSTRACT

Dynamic facial expression recognition (FER) databases provide important data support for affective computing and applications. However, most FER databases are annotated with several basic mutually exclusive emotional categories and contain only one modality, e.g., videos. The monotonous labels and modality cannot accurately imitate human emotions and fulfill applications in the real world. In this paper, we propose MAFW, a large-scale multi-modal compound affective database with 10,045 video-audio clips in the wild. Each clip is annotated with a compound emotional category and a couple of sentences that describe the subjects' affective behaviors in the clip. For the compound emotion annotation, each clip is categorized into one or more of the 11 widely-used emotions, i.e., anger, disgust, fear, happiness, neutral, sadness, surprise, contempt, anxiety, helplessness, and disappointment. To ensure high quality of the labels, we filter out the unreliable annotations by an Expectation Maximization (EM) algorithm, and then obtain 11 single-label emotion categories and 32 multi-label emotion categories. To the best of our knowledge, MAFW is the first in-the-wild multi-modal database annotated with compound emotion annotations and emotion-related captions. Additionally, we also propose a novel Transformer-based expression snippet feature learning method to recognize the compound emotions leveraging the expression-change relations among different emotions and modalities. Extensive experiments on MAFW database show the advantages of the proposed method over other state-of-the-art methods for both uni- and multi-modal FER. Our MAFW database is publicly available from https://mafw-database.github.io/MAFW.

Motivation & Objective

• To address the limitation of existing dynamic facial expression databases that use only basic, mutually exclusive emotions and single modalities.
• To create a large-scale, in-the-wild, multi-modal database with compound emotion annotations reflecting real-world emotional complexity.
• To develop a robust method for recognizing compound emotions by modeling expression-change dynamics across visual, audio, and textual modalities.
• To enable new research in affective computing by providing high-quality, multi-modal, and context-rich data for dynamic FER.

Proposed method

• Proposes MAFW, a large-scale, in-the-wild, multi-modal database with 10,045 video-audio clips and 20,000 textual affective captions.
• Uses an Expectation Maximization (EM) algorithm to filter unreliable annotations, resulting in 11 single-label and 32 multi-label emotion categories.
• Introduces a novel Transformer-based expression snippet feature learning (T-ESFL) method that models expression-change relations across modalities.
• Employs snippet-based Transformer with spatial-spectral order reconstruction (SSOR) for visual feature learning, ResNet_LSTM for audio, and DPCNN for text.
• Combines modality-specific features via concatenation to form a multi-modal emotion representation.
• Optimizes the model using a joint objective combining cross-entropy loss and snippet order reconstruction loss to enhance temporal dynamics modeling.

Experimental results

Research questions

• RQ1 Can a large-scale, in-the-wild, multi-modal database with compound emotion annotations improve the realism and complexity of dynamic facial expression recognition?
• RQ2 How effective is the proposed T-ESFL method in recognizing compound emotions compared to state-of-the-art approaches across uni- and multi-modal settings?
• RQ3 To what extent do expression-change relations across modalities enhance recognition performance in dynamic FER?
• RQ4 Can the MAFW database support downstream tasks such as video emotional captioning with high-quality, context-rich descriptions?

Key findings

• The proposed T-ESFL method achieves state-of-the-art performance on both uni- and multi-modal dynamic facial expression recognition, outperforming existing methods on the MAFW dataset.
• Video emotional captioning using off-the-shelf models achieved BLEU-4 of 9.09, METEOR of 15.49, and CIDEr of 23.40, demonstrating the utility of MAFW for natural language generation tasks.
• The EM-based filtering process successfully reduced annotation noise, yielding 11 single-label and 32 multi-label emotion categories with high reliability.
• Qualitative analysis confirms that the emotional captions generated by the models align well with ground truth, capturing nuanced affective behaviors and contextual cues.
• The MAFW database is publicly available at https://mafw-database.github.io/MAFW, enabling broader research in affective computing.
• The dataset exhibits minimal demographic bias, with gender statistics inferred from CelebA and used only for data distribution analysis, not model training.