[Paper Review] Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention

Ting Wu, Junjie Peng+4 more|arXiv (Cornell University)|Mar 3, 2021

Emotion and Mood RecognitionPsychology66 references119 citations

TL;DR

This paper proposes Bimodal Information-augmented Multi-Head Attention (BIMHA), a novel multimodal fusion framework for video sentiment analysis that models inter-bimodal interactions via attention mechanisms across textual, visual, and acoustic modalities. By assigning dynamic attention weights to acoustic-visual, acoustic-textual, and visual-textual feature pairs and fusing them through a residual structure, BIMHA improves sentiment prediction accuracy across four public datasets, outperforming existing methods with state-of-the-art results on MOSI, MOSEI, and IEMOCAP.

ABSTRACT

Humans express feelings or emotions via different channels. Take language as an example, it entails different sentiments under different visual-acoustic contexts. To precisely understand human intentions as well as reduce the misunderstandings caused by ambiguity and sarcasm, we should consider multimodal signals including textual, visual and acoustic signals. The crucial challenge is to fuse different modalities of features for sentiment analysis. To effectively fuse the information carried by different modalities and better predict the sentiments, we design a novel multi-head attention based fusion network, which is inspired by the observations that the interactions between any two pair-wise modalities are different and they do not equally contribute to the final sentiment prediction. By assigning the acoustic-visual, acoustic-textual and visual-textual features with reasonable attention and exploiting a residual structure, we attend to attain the significant features. We conduct extensive experiments on four public multimodal datasets including one in Chinese and three in English. The results show that our approach outperforms the existing methods and can explain the contributions of bimodal interaction in multiple modalities.

Motivation & Objective

• To address the challenge of multimodal sentiment fusion where different modality pairs contribute unequally to sentiment prediction.
• To model intra-modal, inter-modal, and inter-bimodal interactions simultaneously for richer feature representation.
• To improve sentiment prediction by dynamically weighting bimodal interactions using an extended multi-head attention mechanism.
• To provide visual explanations of how different bimodal combinations (AV, AT, VT) contribute to sentiment decisions across samples.

Proposed method

• Proposes Bimodal Multi-Head Attention (BMHA), an extension of multi-head attention that uses multimodal features as queries and bimodal features as keys and values.
• Applies tensor fusion to generate bimodal interaction features before applying BMHA for dynamic attention weighting.
• Uses residual connections to preserve original inter-modal features while fusing them with attention-enhanced bimodal features.
• Employs three parallel MHA heads for acoustic-visual, acoustic-textual, and visual-textual interactions, each learning modality-specific attention patterns.
• Integrates the weighted bimodal features into a sentiment inference network for final prediction.
• Visualizes attention scores across samples to interpret the contribution of each bimodal pair in real-time decision-making.

Experimental results

Research questions

• RQ1 How do different bimodal interactions (AV, AT, VT) contribute variably to sentiment prediction across video samples?
• RQ2 Can a multi-head attention mechanism effectively model inter-bimodal interactions while preserving unimodal and inter-modal representations?
• RQ3 Does dynamic attention weighting across bimodal feature pairs improve sentiment classification performance compared to fixed fusion strategies?
• RQ4 To what extent can the proposed model explain its predictions through attention visualization?
• RQ5 How does the model generalize across diverse multimodal datasets, including low-resource settings like Chinese video sentiment analysis?

Key findings

• BIMHA achieves state-of-the-art performance on the CMU-MOSI dataset, with a test accuracy of 83.44% (Acc-2) and F1 score of 85.46% on the 'negative/non-negative' classification.
• On the MOSEI dataset, BIMHA achieves 83.19% accuracy (Acc-2) and 83.21% F1 score for 'negative/positive' sentiment classification, outperforming prior methods.
• On the IEMOCAP dataset, BIMHA achieves 86.57% accuracy and 85.8% F1 for the 'happy' class, demonstrating strong performance across emotion categories.
• Attention visualization shows that VT (visual-textual) features contribute most consistently across the dataset, while AV (audio-visual) features dominate in specific samples.
• The model's performance improves with the introduction of unaligned data training, achieving 53.87% Acc-2 and 0.765 Corr on Self-MM's unaligned setting, indicating robustness to data distribution shifts.
• The ablation study confirms that incorporating bimodal attention significantly enhances performance, with BIMHA2 (unified attention) outperforming BIMHA1 (aligned) on multiple metrics.