[Paper Review] A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition

Ziwang Fu, Feng Liu+5 more|arXiv (Cornell University)|Nov 3, 2021

Emotion and Mood RecognitionPsychology25 references28 citations

TL;DR

CFN-SR proposes a cross-modal fusion network using self-attention and a residual design to fuse audio and video features, achieving 75.76% accuracy on RAVDESS with 26.30M params.

ABSTRACT

The audio-video based multimodal emotion recognition has attracted a lot of attention due to its robust performance. Most of the existing methods focus on proposing different cross-modal fusion strategies. However, these strategies introduce redundancy in the features of different modalities without fully considering the complementary properties between modal information, and these approaches do not guarantee the non-loss of original semantic information during intra- and inter-modal interactions. In this paper, we propose a novel cross-modal fusion network based on self-attention and residual structure (CFN-SR) for multimodal emotion recognition. Firstly, we perform representation learning for audio and video modalities to obtain the semantic features of the two modalities by efficient ResNeXt and 1D CNN, respectively. Secondly, we feed the features of the two modalities into the cross-modal blocks separately to ensure efficient complementarity and completeness of information through the self-attention mechanism and residual structure. Finally, we obtain the output of emotions by splicing the obtained fused representation with the original representation. To verify the effectiveness of the proposed method, we conduct experiments on the RAVDESS dataset. The experimental results show that the proposed CFN-SR achieves the state-of-the-art and obtains 75.76% accuracy with 26.30M parameters. Our code is available at https://github.com/skeletonNN/CFN-SR.

Motivation & Objective

• Motivate robust multimodal emotion recognition from audio and video data.
• Develop a fusion framework that preserves original semantic information during intra- and inter-modal interactions.
• Leverage self-attention for audio feature selection and a residual pathway to retain video semantics.
• Ensure efficient complementarity between modalities through cross-modal blocks.

Proposed method

• Extract video features with 3D ResNeXt50 for spatio-temporal representation.
• Extract audio features with a simple 1D CNN on MFCCs.
• Apply cross-modal blocks where audio undergoes intra-modal self-attention before inter-modal interaction with video.
• Fuse modalities via a tanh-based inter-modal mapping and softmax-guided fusion while preserving video integrity through a residual structure.
• Concatenate fused representation with original features and classify with cross-entropy loss.

Experimental results

Research questions

• RQ1 How can cross-modal blocks enable efficient and comprehensive interaction between audio and video features for emotion recognition?
• RQ2 Does incorporating self-attention for audio feature selection and a residual path for video features improve recognition accuracy and information integrity?
• RQ3 What is the impact of fusion direction (V->A vs A->V) on performance in this architecture?
• RQ4 How does CFN-SR compare to existing fusion approaches on standard multimodal emotion datasets?

Key findings

• Achieves 75.76% accuracy on RAVDESS with 26.30M parameters, attaining state-of-the-art performance.
• Cross-modal blocks improve accuracy by over 2% with a modest parameter increase (~0.4M).
• Self-attention-driven audio feature selection contributes significantly to performance, reducing redundancy.
• Residual structure helps preserve video modality information during inter-modal interactions, improving robustness.
• A->V fusion yields better performance than V->A in this setup, indicating effective audio feature selection aids inter-modal interaction.