[论文解读] Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention

R. Gnana Praveen, Éric Granger+1 more|arXiv (Cornell University)|Sep 19, 2022

Emotion and Mood Recognition被引用 4

一句话总结

本文提出了一种用于维度情感识别中视听融合的联合交叉注意力（JCA）机制，通过基于联合模态与单个模态特征之间相关性的计算，建模模态内与模态间的关系。该方法在RECOLA和AffWild2数据集上优于最先进模型，尤其在音频噪声大或缺失的情况下表现出更强的鲁棒性，并提升了愉悦度与唤醒度预测的性能。

ABSTRACT

Automatic emotion recognition (ER) has recently gained lot of interest due to its potential in many real-world applications. In this context, multimodal approaches have been shown to improve performance (over unimodal approaches) by combining diverse and complementary sources of information, providing some robustness to noisy and missing modalities. In this paper, we focus on dimensional ER based on the fusion of facial and vocal modalities extracted from videos, where complementary audio-visual (A-V) relationships are explored to predict an individual's emotional states in valence-arousal space. Most state-of-the-art fusion techniques rely on recurrent networks or conventional attention mechanisms that do not effectively leverage the complementary nature of A-V modalities. To address this problem, we introduce a joint cross-attentional model for A-V fusion that extracts the salient features across A-V modalities, that allows to effectively leverage the inter-modal relationships, while retaining the intra-modal relationships. In particular, it computes the cross-attention weights based on correlation between the joint feature representation and that of the individual modalities. By deploying the joint A-V feature representation into the cross-attention module, it helps to simultaneously leverage both the intra and inter modal relationships, thereby significantly improving the performance of the system over the vanilla cross-attention module. The effectiveness of our proposed approach is validated experimentally on challenging videos from the RECOLA and AffWild2 datasets. Results indicate that our joint cross-attentional A-V fusion model provides a cost-effective solution that can outperform state-of-the-art approaches, even when the modalities are noisy or absent.

研究动机与目标

• 通过有效融合视频数据中的音频与视觉模态，提升愉悦度-唤醒度空间中的维度情感识别性能。
• 解决现有融合方法无法联合建模视听特征中模态内与模态间关系的局限性。
• 开发一种成本低、鲁棒性强的融合机制，即使在音频噪声大或缺失时仍能保持高性能。
• 在具有多样化情感表达和挑战性条件的真实世界数据集上验证所提方法的有效性。

提出的方法

• 该方法采用独立的主干网络分别提取音频（A）和视觉（V）模态的初始特征。
• 通过联合A-V特征表示与单个A和V特征之间的相关性，计算联合交叉注意力权重。
• 注意力机制使每个模态能够关注另一模态以及自身的模态内表示，从而同时保留模态间与模态内关系。
• 将两个模态的注意力加权特征拼接后，通过全连接层预测愉悦度与唤醒度分数。
• 模型采用端到端方式进行训练，并在多种条件下进行评估，包括音频段缺失的情况。
• 该方法设计为模块化结构，可兼容不同架构的A和V模态主干网络。

Figure 1: The valence-arousal space. Valence denotes the range of emotions from being very sad (negative) to very happy (positive) and arousal reflects the energy or intensity of emotions from very passive to very active.

实验结果

研究问题

• RQ1 联合建模模态内与模态间关系是否能提升视听融合在维度情感识别中的性能？
• RQ2 与原始交叉注意力相比，基于联合与单个模态特征之间相关性的注意力机制在特征表示上有哪些改进？
• RQ3 当音频模态存在噪声或缺失时，所提方法在多大程度上仍能保持性能？
• RQ4 该模型在情绪预测过程中如何应对时间变化、面部遮挡和姿态变化？
• RQ5 与先前基于注意力的融合方法相比，联合交叉注意力机制是否能生成更准确、更具上下文感知能力的注意力图？

主要发现

• 所提出的联合交叉注意力（JCA）模型在RECOLA和AffWild2数据集上预测愉悦度与唤醒度的性能均优于最先进方法。
• 即使在音频段缺失或存在噪声时，该模型仍表现出卓越性能，展现出对模态退化较强的鲁棒性。
• 可视化结果表明，JCA能有效聚焦于相关面部表情与语音能量变化，而原始交叉注意力常会遗漏关键时间片段。
• 该模型在姿态变化或部分面部遮挡情况下，对真实愉悦度与唤醒度的跟踪更为稳定。
• 尽管性能优异，该模型在唤醒度预测上仍略低于愉悦度，尤其在模糊或姿态变化导致注意力机制误判时更为明显。
• 注意力机制成功捕捉了音频与视觉模态之间的上下文关系，从而实现更准确且时间上一致的预测。

Figure 2: Joint cross-attention model proposed for A-V fusion (in testing mode).