[Paper Review] A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition

Yue Jin, Tianqing Zheng+2 more|arXiv (Cornell University)|Jul 9, 2021

Emotion and Mood Recognition22 references21 citations

TL;DR

This paper proposes a multi-modal, multi-task learning framework that jointly recognizes facial Action Units (AUs) and expressions using both visual and audio inputs. By training a shared visual backbone with AU and expression heads, freezing it, and fusing visual and audio features through a transformer encoder, the method achieves an AU recognition score of 0.712 and expression recognition score of 0.477 on the Aff-Wild2 validation set, significantly outperforming the baseline.

ABSTRACT

Analyzing human affect is vital for human-computer interaction systems. Most methods are developed in restricted scenarios which are not practical for in-the-wild settings. The Affective Behavior Analysis in-the-wild (ABAW) 2021 Contest provides a benchmark for this in-the-wild problem. In this paper, we introduce a multi-modal and multi-task learning method by using both visual and audio information. We use both AU and expression annotations to train the model and apply a sequence model to further extract associations between video frames. We achieve an AU score of 0.712 and an expression score of 0.477 on the validation set. These results demonstrate the effectiveness of our approach in improving model performance.

Motivation & Objective

• To address the challenge of affect recognition in real-world, in-the-wild video settings where data is noisy and imbalanced.
• To improve performance in Action Unit (AU) and facial expression recognition by leveraging both visual and audio modalities.
• To mitigate class imbalance in AU and expression datasets through task-specific loss functions and multi-task training.
• To enhance temporal modeling of facial sequences using a transformer-based encoder for long-range frame dependencies.
• To ensure fair evaluation by preventing data leakage between AU and expression validation sets.

Proposed method

• A multi-task visual backbone is trained on Glint360K using Cosface loss to extract robust face features, with shared weights for AU and expression heads.
• AU recognition uses BCE loss with class weights to handle positive sample imbalance, while expression recognition uses focal loss to emphasize hard examples.
• The visual model is frozen after pretraining, and an audio stream is added using a TDNN-based model on Mel-spectrograms to extract audio features.
• Visual and audio features are concatenated and fed into a single-layer transformer encoder to model temporal dynamics across video frames.
• Training alternates between AU and expression heads (epoch- or batch-wise) to handle incomplete annotations in the Aff-Wild2 dataset.
• The final prediction head is a fully connected layer that outputs per-frame AU and expression probabilities.

Experimental results

Research questions

• RQ1 Can joint multi-task learning of AUs and expressions improve performance on in-the-wild affect recognition?
• RQ2 Does incorporating audio modality enhance AU and expression recognition beyond visual cues alone?
• RQ3 How effective is a transformer-based sequence model in capturing temporal dependencies in facial sequences for affect recognition?
• RQ4 To what extent can class imbalance in AU and expression datasets be mitigated using BCE and focal loss respectively?
• RQ5 Can a multi-modal, multi-task framework reduce overfitting and improve generalization in real-world video settings?

Key findings

• The proposed method achieves an AU recognition score of 0.712 on the Aff-Wild2 validation set, significantly outperforming the baseline score of 0.31.
• For expression recognition, the method achieves a score of 0.477, compared to the baseline of 0.366, demonstrating substantial improvement.
• The use of both visual and audio modalities leads to better performance than relying on visual data alone, especially in challenging in-the-wild conditions.
• The multi-task training strategy with alternating backpropagation effectively handles incomplete annotations in the AU and expression datasets.
• The transformer encoder effectively models long-range temporal dependencies, improving sequence-level recognition accuracy.
• The ablation of data leakage by removing overlapping validation samples ensures a fair and reliable evaluation of model performance.