[论文解读] A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition

Ziwang Fu, Feng Liu+5 more|arXiv (Cornell University)|Nov 3, 2021

Emotion and Mood Recognition参考文献 25被引用 28

一句话总结

CFN-SR 提出了一个使用自注意力和残差设计的跨模态融合网络，用于融合音频和视频特征，在 RAVDESS 数据集上达到 75.76% 的准确率，参数量为 26.30M。

ABSTRACT

The audio-video based multimodal emotion recognition has attracted a lot of attention due to its robust performance. Most of the existing methods focus on proposing different cross-modal fusion strategies. However, these strategies introduce redundancy in the features of different modalities without fully considering the complementary properties between modal information, and these approaches do not guarantee the non-loss of original semantic information during intra- and inter-modal interactions. In this paper, we propose a novel cross-modal fusion network based on self-attention and residual structure (CFN-SR) for multimodal emotion recognition. Firstly, we perform representation learning for audio and video modalities to obtain the semantic features of the two modalities by efficient ResNeXt and 1D CNN, respectively. Secondly, we feed the features of the two modalities into the cross-modal blocks separately to ensure efficient complementarity and completeness of information through the self-attention mechanism and residual structure. Finally, we obtain the output of emotions by splicing the obtained fused representation with the original representation. To verify the effectiveness of the proposed method, we conduct experiments on the RAVDESS dataset. The experimental results show that the proposed CFN-SR achieves the state-of-the-art and obtains 75.76% accuracy with 26.30M parameters. Our code is available at https://github.com/skeletonNN/CFN-SR.

研究动机与目标

• 推动基于音频和视频数据的鲁棒多模态情感识别。
• 开发一个融合框架，在模态内与模态间交互过程中保留原始语义信息。
• 利用自注意力进行音频特征选择，并通过残差路径保留视频语义。
• 通过跨模态块确保模态之间的高效互补。

提出的方法

• 使用 3D ResNeXt50 提取视频特征以获得时空表示。
• 在 MFCCs 上用一个简单的一维卷积神经网络提取音频特征。
• 应用跨模态块，其中音频在与视频进行跨模态交互之前先进行模态内自注意力。
• 通过基于 tanh 的跨模态映射和 softmax 指导的融合来融合模态，并通过残差结构保持视频的完整性。
• 将融合表示与原始特征连接，并使用交叉熵损失进行分类。

实验结果

研究问题

• RQ1 跨模态块如何实现音频和视频特征在情感识别中的高效且全面的交互？
• RQ2 将自注意力用于音频特征选择以及为视频特征设置残差路径是否能提高识别准确性和信息完整性？
• RQ3 在该架构中，融合方向（V->A 与 A->V）对性能有何影响？
• RQ4 CFN-SR 与现有的融合方法在标准多模态情感数据集上的表现如何？

主要发现

• 在 RAVDESS 上达到 75.76% 的准确率，参数量为 26.30M，达到最先进的性能。
• 跨模态块在参数增加约 0.4M 的情况下，准确性提升超过 2%。
• 以自注意力驱动的音频特征选择对性能贡献显著，降低冗余。
• 残差结构有助于在跨模态交互中保留视频模态信息，提升鲁棒性。
• 在该设置中 A->V 融合优于 V->A，表明有效的音频特征选择有助于跨模态交互。