[Paper Review] Multi-modal Expression Recognition with Ensemble Method

Chuanhe Liu, Xinjie Zhang+6 more|arXiv (Cornell University)|Mar 17, 2023

Speech and Audio Processing4 citations

TL;DR

This paper presents an ensemble-based multi-modal expression recognition system for the ABAW 2023 challenge, combining diverse pre-trained visual and audio features with temporal encoders (LSTM and Transformer) to improve emotion recognition. The method achieves an average F1 score of 0.45774 on the validation set through strategic feature fusion and model ensemble strategies.

ABSTRACT

This paper presents our submission to the Expression Classification Challenge of the fifth Affective Behavior Analysis in-the-wild (ABAW) Competition. In our method, multimodal feature combinations extracted by several different pre-trained models are applied to capture more effective emotional information. For these combinations of visual and audio modal features, we utilize two temporal encoders to explore the temporal contextual information in the data. In addition, we employ several ensemble strategies for different experimental settings to obtain the most accurate expression recognition results. Our system achieves the average F1 Score of 0.45774 on the validation set.

Motivation & Objective

• To improve affective behavior analysis in the wild by leveraging multi-modal data (visual and audio) for more robust expression recognition.
• To address the limitations of unimodal models by combining complementary visual and audio features to enhance model generalization and robustness.
• To explore effective feature combinations from multiple pre-trained models across visual and audio modalities.
• To optimize temporal modeling of sequential emotional dynamics using both LSTM and Transformer architectures.
• To improve final prediction performance through structured model ensemble strategies across diverse architectures and feature sets.

Proposed method

• Pre-process video data by extracting and aligning face images using a face detector, with temporal imputation for missing frames.
• Extract multi-modal features using a suite of pre-trained models: DenseNet, IResNet100, MobileNet, MAE for visual features; eGeMAPS, ComParE 2016, VGGish, Wav2Vec 2.0, ECAPA-TDNN, and HuBERT for audio features.
• Construct multimodal feature representations by concatenating diverse visual and audio feature embeddings from different pre-trained models.
• Apply two temporal encoders—LSTM and Transformer—on the concatenated multimodal features to model sequential context and temporal dependencies.
• Train models using Adam optimizer with a 25-epoch schedule, tuning hyperparameters such as dropout (0.3), attention heads (4), and sequence length (128).
• Employ ensemble strategies, including voting across models with different architectures and feature combinations, to boost final prediction accuracy.

Experimental results

Research questions

• RQ1 How do different combinations of pre-trained visual and audio features impact expression recognition performance?
• RQ2 What is the comparative effectiveness of LSTM versus Transformer architectures in modeling temporal dynamics of facial and audio expressions?
• RQ3 To what extent does model ensembling improve robustness and accuracy in multi-modal expression recognition?
• RQ4 How do self-supervised visual features (e.g., MAE) compare to supervised models (e.g., IResNet100) in the context of expression recognition?
• RQ5 What is the contribution of hand-crafted vs. deep neural network-based audio features in multi-modal emotion recognition?

Key findings

• The ensemble of multiple models with diverse feature combinations and architectures achieved the highest validation F1 score of 0.45774.
• The best-performing individual model (LSTM with densenet, mae, ires100, ecapatdnn, hubert) achieved an F1 score of 0.41410.
• The Transformer-based model with mae, ires100, densenet, ecapatdnn, hubert features achieved an F1 score of 0.39380.
• In ablation studies, combining MAE, IResNet100, and DenseNet visual features with ecapatdnn and hubert audio features yielded the highest F1 (0.39380).
• The inclusion of hand-crafted features (e.g., ecapatdnn, fbank) improved performance over using only deep learning-based audio features in some configurations.
• The model ensemble strategy significantly outperformed individual models, demonstrating the value of diversity in model architecture and feature selection.