[Paper Review] Cross-Modality Gated Attention Fusion for Multimodal Sentiment Analysis

Ming Jiang, Shaoxiong Ji|arXiv (Cornell University)|Aug 25, 2022

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper proposes CMGA, a Cross-Modality Gated Attention fusion model for multimodal sentiment analysis that uses cross-attention between modality pairs and a forget gate to filter noisy signals. The method achieves state-of-the-art performance on MOSI and MOSEI datasets by enhancing cross-modality interaction while preserving unique modality-specific features.

ABSTRACT

Multimodal sentiment analysis is an important research task to predict the sentiment score based on the different modality data from a specific opinion video. Many previous pieces of research have proved the significance of utilizing the shared and unique information across different modalities. However, the high-order combined signals from multimodal data would also help extract satisfied representations. In this paper, we propose CMGA, a Cross-Modality Gated Attention fusion model for MSA that tends to make adequate interaction across different modality pairs. CMGA also adds a forget gate to filter the noisy and redundant signals introduced in the interaction procedure. We experiment on two benchmark datasets in MSA, MOSI, and MOSEI, illustrating the performance of CMGA over several baseline models. We also conduct the ablation study to demonstrate the function of different components inside CMGA.

Motivation & Objective

• To address the challenge of noisy and redundant signals introduced during multimodal interaction in sentiment analysis.
• To enhance cross-modality feature fusion by explicitly modeling interactions between all modality pairs (text–video, video–audio, text–audio).
• To preserve unique modality-specific representations while effectively integrating shared signals across modalities.
• To improve sentiment prediction performance through a forget gate mechanism that filters irrelevant or redundant attention features.
• To validate the effectiveness of component design via ablation studies on modality importance and architectural modules.

Proposed method

• The model uses pre-trained BERT for text, bi-LSTM with fully connected layers for visual and audio features, all projected to a shared dimension $ d_k $.
• Cross-modality attention is applied to three modality pairs: (text, video), (video, audio), and (text, audio), using query, key, and value matrices derived from paired modalities.
• For each pair, the attention mechanism computes weighted interactions where one modality acts as query and the other as key/value, enabling cross-modal alignment.
• A forget gate is applied to the cross-attention output to suppress noisy or redundant features, improving signal-to-noise ratio in fused representations.
• Residual connections are used to preserve original modality features and prevent degradation during training.
• A transformer-based fusion layer aggregates the refined cross-attention features to predict the final sentiment score.

Experimental results

Research questions

• RQ1 How does the proposed CMGA model improve sentiment prediction accuracy compared to existing multimodal fusion methods?
• RQ2 What is the contribution of the forget gate mechanism in filtering redundant or noisy signals during cross-modality attention?
• RQ3 How do different modality interaction orders (e.g., (text, video) vs. (video, text)) affect model performance?
• RQ4 What is the individual contribution of each modality (text, video, audio) to the final prediction, and how does removing them affect performance?
• RQ5 How do architectural components like cross-attention and residual connections influence the model’s effectiveness?

Key findings

• CMGA achieves a MAE of 0.790 and Acc-7 of 43.29 on the MOSI dataset, outperforming baseline models in most evaluation metrics.
• On the MOSEI dataset, CMGA achieves a MAE of 0.545 and Acc-7 of 53.03, demonstrating strong generalization across datasets.
• Ablation studies show that removing the forget gate increases MAE to 0.856 on MOSI and 0.594 on MOSEI, confirming its role in noise suppression.
• Removing the cross-attention mechanism leads to a significant performance drop, with MAE rising to 0.845 on MOSI and 0.587 on MOSEI, indicating its importance in cross-modality learning.
• The ablation study reveals that the textual modality contributes most significantly to performance, as removing it causes the largest drop in accuracy (to 30.51% on MOSI).
• Reversing the modality pair order shows that text-to-video attention is more critical than video-to-audio, as performance drops more when the text modality is used as the key (e.g., from 0.790 to 0.814 MAE on MOSI when reversing (text, video) to (video, text)).