[论文解读] Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment Analysis

Feiyang Chen, Ziqian Luo+2 more|arXiv (Cornell University)|Apr 17, 2019

Sentiment Analysis and Opinion Mining参考文献 35被引用 53

一句话总结

引入 DFF-ATMF，一种双分支音频-文本多模态模型，通过将多特征音频融合与多模态注意力相结合，提升在 CMU-MOSI、CMU-MOSEI 和 IEMOCAP 数据集上的情感分析性能。它实现了具有竞争力的或最先进的结果，并展示了强大的特征互补性和鲁棒性。

ABSTRACT

Sentiment analysis, mostly based on text, has been rapidly developing in the last decade and has attracted widespread attention in both academia and industry. However, the information in the real world usually comes from multiple modalities, such as audio and text. Therefore, in this paper, based on audio and text, we consider the task of multimodal sentiment analysis and propose a novel fusion strategy including both multi-feature fusion and multi-modality fusion to improve the accuracy of audio-text sentiment analysis. We call it the DFF-ATMF (Deep Feature Fusion - Audio and Text Modality Fusion) model, which consists of two parallel branches, the audio modality based branch and the text modality based branch. Its core mechanisms are the fusion of multiple feature vectors and multiple modality attention. Experiments on the CMU-MOSI dataset and the recently released CMU-MOSEI dataset, both collected from YouTube for sentiment analysis, show the very competitive results of our DFF-ATMF model. Furthermore, by virtue of attention weight distribution heatmaps, we also demonstrate the deep features learned by using DFF-ATMF are complementary to each other and robust. Surprisingly, DFF-ATMF also achieves new state-of-the-art results on the IEMOCAP dataset, indicating that the proposed fusion strategy also has a good generalization ability for multimodal emotion recognition.

研究动机与目标

• 在文本之外利用音频来推动多模态情感分析。
• 提出一种融合策略，将多特征融合与多模态融合相结合。
• 在 CMU-MOSI、CMU-MOSEI 和 IEMOCAP 数据集上开发并评估 DFF-ATMF 模型。

提出的方法

• 为音频和文本模态设有两个并行分支，采用基于 Bi-LSTM 的特征提取。
• 在每个模态内部进行多特征融合（结合原始波形和声学特征）。
• 使用 BERT 嵌入的文本表示，随后进行 Bi-LSTM 和注意力以获得 TSV（Text Sentiment Vector）。
• 多模态注意力融合，将 ASV 与 TSV及其多特征向量结合用于最终预测。
• 以交叉熵损失、 dropout 和 Adam 优化器进行训练；评估采用加权准确率和 Macro F1。

实验结果

研究问题

• RQ1 结合多特征与多模态注意力的融合策略是否能在多模态情感分析中超越单特征或单模态基线？
• RQ2 从提出的 DFF-ATMF 模型学习的特征，在数据集与任务（情感识别和情绪识别）上是否表现出互补性和鲁棒性？

主要发现

• DFF-ATMF 在提出的设置中在 CMU-MOSI 上达到 80.98% 的准确率和 81.26% 的 F1。
• DFF-ATMF 在 CMU-MOSEI 上达到 77.15% 的准确率和 78.33% 的 F1。
• 在 IEMOCAP 上，DFF-ATMF 获得 81.37% 的总体准确率和 82.29% 的 Macro F1。
• DFF-ATMF 在 CMU-MOSI 和 IEMOCAP 上超越若干最先进的模型，在 CMU-MOSEI 上也具有竞争力。
• 注意力热图显示学习的特征在数据集之间具有互补性和鲁棒性。