[Paper Review] Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment Analysis

Feiyang Chen, Ziqian Luo+2 more|arXiv (Cornell University)|Apr 17, 2019

Sentiment Analysis and Opinion MiningComputer Science35 references53 citations

TL;DR

Introduces DFF-ATMF, a dual-branch audio-text multimodal model that merges multi-feature audio fusion with multimodal attention to improve sentiment analysis across CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets. It achieves competitive or state-of-the-art results and demonstrates strong feature complementarity and robustness.

ABSTRACT

Sentiment analysis, mostly based on text, has been rapidly developing in the last decade and has attracted widespread attention in both academia and industry. However, the information in the real world usually comes from multiple modalities, such as audio and text. Therefore, in this paper, based on audio and text, we consider the task of multimodal sentiment analysis and propose a novel fusion strategy including both multi-feature fusion and multi-modality fusion to improve the accuracy of audio-text sentiment analysis. We call it the DFF-ATMF (Deep Feature Fusion - Audio and Text Modality Fusion) model, which consists of two parallel branches, the audio modality based branch and the text modality based branch. Its core mechanisms are the fusion of multiple feature vectors and multiple modality attention. Experiments on the CMU-MOSI dataset and the recently released CMU-MOSEI dataset, both collected from YouTube for sentiment analysis, show the very competitive results of our DFF-ATMF model. Furthermore, by virtue of attention weight distribution heatmaps, we also demonstrate the deep features learned by using DFF-ATMF are complementary to each other and robust. Surprisingly, DFF-ATMF also achieves new state-of-the-art results on the IEMOCAP dataset, indicating that the proposed fusion strategy also has a good generalization ability for multimodal emotion recognition.

Motivation & Objective

• Motivate multimodal sentiment analysis beyond text by leveraging audio in addition to text.
• Propose a fusion strategy that combines multi-feature fusion with multi-modality fusion.
• Develop and evaluate the DFF-ATMF model on CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets.

Proposed method

• Two parallel branches for audio and text modalities with Bi-LSTM based feature extraction.
• Multi-feature fusion within each modality (combining raw waveforms and acoustic features).
• Text representation using BERT embeddings followed by Bi-LSTM and attention for TSV (Text Sentiment Vector).
• Multimodal-attention fusion combining ASV and TSV with their multi-feature vectors for final prediction.
• Training with cross-entropy loss, dropout, and Adam optimizer; evaluation with weighted accuracy and Macro F1.

Experimental results

Research questions

• RQ1 Can a fusion strategy that combines multi-feature and multi-modality attention improve multimodal sentiment analysis over single-feature or single-modality baselines?
• RQ2 Do learned features from the proposed DFF-ATMF model exhibit complementarity and robustness across datasets and tasks (sentiment and emotion recognition)?

Key findings

• DFF-ATMF achieves 80.98% accuracy (CMU-MOSI) and 81.26% F1 (CMU-MOSI) on the proposed setup.
• DFF-ATMF reaches 77.15% accuracy (CMU-MOSEI) and 78.33% F1 (CMU-MOSEI).
• On IEMOCAP, DFF-ATMF attains 81.37% overall accuracy and 82.29% Macro F1.
• DFF-ATMF outperforms several state-of-the-art models on CMU-MOSI and IEMOCAP and is competitive on CMU-MOSEI.
• Attention heatmaps indicate learned features are complementary and robust across datasets.