[论文解读] Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition

Yuntao Shou, Tao Meng+3 more|arXiv (Cornell University)|Dec 28, 2023

Emotion and Mood Recognition被引用 4

一句话总结

该论文提出 AR-IIGCN，一种新颖的多模态情感识别框架，通过对抗性表示学习减少模态异质性，并通过图对比学习捕捉模态内、模态间、类内和类间语义关系。该方法在 IEMOCAP 和 MELD 数据集上实现了最先进性能，在使用 RoBERTa-Large 作为文本编码器时，IEMOCAP 数据集上的加权 F1 达到 70.4%，通过联合模态对齐与对比学习，展现出优越的特征表示与分类准确率。

ABSTRACT

With the release of increasing open-source emotion recognition datasets on social media platforms and the rapid development of computing resources, multimodal emotion recognition tasks (MER) have begun to receive widespread research attention. The MER task extracts and fuses complementary semantic information from different modalities, which can classify the speaker's emotions. However, the existing feature fusion methods have usually mapped the features of different modalities into the same feature space for information fusion, which can not eliminate the heterogeneity between different modalities. Therefore, it is challenging to make the subsequent emotion class boundary learning. To tackle the above problems, we have proposed a novel Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive for Multimodal Emotion Recognition (AR-IIGCN) method. Firstly, we input video, audio, and text features into a multi-layer perceptron (MLP) to map them into separate feature spaces. Secondly, we build a generator and a discriminator for the three modal features through adversarial representation, which can achieve information interaction between modalities and eliminate heterogeneity among modalities. Thirdly, we introduce contrastive graph representation learning to capture intra-modal and inter-modal complementary semantic information and learn intra-class and inter-class boundary information of emotion categories. Specifically, we construct a graph structure for three modal features and perform contrastive representation learning on nodes with different emotions in the same modality and the same emotion in different modalities, which can improve the feature representation ability of nodes. Extensive experimental works show that the ARL-IIGCN method can significantly improve emotion recognition accuracy on IEMOCAP and MELD datasets.

研究动机与目标

• 为解决多模态情感识别中的模态异质性挑战，即文本、语音和视频特征被映射到共享空间时导致的对齐偏差与边界学习不佳问题。
• 通过捕捉模态间与模态内互补的语义信息，提升特征表示能力。
• 通过对比学习显式建模类内紧凑性与类间可分性，增强类别边界学习。
• 开发一种可泛化的框架，适用于情感识别以外的任务，如多模态幽默检测。
• 通过消融研究与跨数据集评估，验证对抗学习与图对比学习组件的有效性。

提出的方法

• 模型使用独立编码器——RoBERTa 用于文本，DenseNet 用于视频，Bi-LSTM 用于语音——提取模态特异性特征，并将其映射到不同的初始特征空间。
• 引入对抗性表示学习模块（TGAN），通过最小化分布差异来对齐不同模态的特征，降低模态异质性。
• 构建图对比学习框架，其中节点代表样本，边在相同模态且相同情感的样本之间（模态内）以及不同模态但相同情感的样本之间（模态间）形成。
• 模态内对比学习最大化同一模态内相同情感样本之间的相似性，而模态间对比学习则最小化跨模态相同情感样本之间的距离。
• 模型采用图神经网络，基于构建的图结构传播并优化表示，增强节点级特征学习。
• 最终通过一个多层感知机（MLP）对优化后的多模态表示进行端到端情感分类。

Figure 1: Illustrative example of the effect of different feature fusion methods on sentiment classification. (a) The feature extraction process for text, video and audio modalities. (b) The model learns emotion class boundaries in a feature fusion manner to map a common feature space. (c) The model

实验结果

研究问题

• RQ1 对抗性表示学习能否有效降低多模态情感识别中的模态异质性？
• RQ2 联合模态内与模态间图对比学习如何改善特征表示与类别边界学习？
• RQ3 对抗性对齐、模态内对比学习与模态间对比学习对整体性能的相对贡献如何？
• RQ4 所提出的 AR-IIGCN 框架能否泛化至情感识别以外的多模态任务？
• RQ5 该模型是否在 IEMOCAP 与 MELD 等基准数据集上达到最先进性能？

主要发现

• 当使用 RoBERTa-Large 作为文本编码器时，AR-IIGCN 在 IEMOCAP 数据集上取得 70.4% 的加权 F1 分数，优于基线模型与消融变体。
• 在 MELD 数据集上，模型使用 RoBERTa-Large 时取得 64.0% 的加权 F1 分数，展现出在不同数据集间的强泛化能力。
• 消融研究显示，模态内与模态间对比学习（IMCL 与 ICCL）对性能贡献最大，其中 IMCL 的性能仅次于完整模型。
• 对抗性对齐（TGAN）组件相较于基线仅带来适度提升，表明其在模态对齐中的作用有限，但影响不及对比学习。
• 该方法在其他多模态任务中也表现出良好泛化能力，在 UR-FUNNY 数据集上应用于 C-MFN 时，幽默检测准确率提升 7.57%。
• 完整 AR-IIGCN 模型在 IEMOCAP 与 MELD 上均持续优于所有消融变体与基线模型，证实其各组件之间的协同效应。

Figure 2: The overall framework of the Adversarial Representation Learning with Intra-Modal and Inter-Modal Graph Contrastive Learning consists of a data preprocessing layer, a multimodal feature fusion layer, a graph contrastive representation learning layer, and an emotion classification layer.