[Paper Review] WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge

Wenbin Wang, Liang Ding+4 more|arXiv (Cornell University)|Jan 12, 2024

Sentiment Analysis and Opinion Mining4 citations

TL;DR

WisdoM proposes a plug-in framework that enhances multimodal sentiment analysis by fusing contextual world knowledge extracted from large vision-language models (LVLMs). It generates relevant context via prompt templates and a training-free Contextual Fusion mechanism, achieving an average F1 improvement of +1.89 over SOTA methods, with gains up to +5.8 F1 on hard samples.

ABSTRACT

Sentiment analysis is rapidly advancing by utilizing various data modalities (e.g., text, image). However, most previous works relied on superficial information, neglecting the incorporation of contextual world knowledge (e.g., background information derived from but beyond the given image and text pairs) and thereby restricting their ability to achieve better multimodal sentiment analysis (MSA). In this paper, we proposed a plug-in framework named WisdoM, to leverage the contextual world knowledge induced from the large vision-language models (LVLMs) for enhanced MSA. WisdoM utilizes LVLMs to comprehensively analyze both images and corresponding texts, simultaneously generating pertinent context. To reduce the noise in the context, we also introduce a training-free contextual fusion mechanism. Experiments across diverse granularities of MSA tasks consistently demonstrate that our approach has substantial improvements (brings an average +1.96% F1 score among five advanced methods) over several state-of-the-art methods.

Motivation & Objective

• Address the limitation of existing multimodal sentiment analysis (MSA) models in lacking contextual world knowledge beyond raw image-text pairs.
• Investigate whether incorporating external, background world knowledge—especially historical or cultural context—can improve sentiment prediction accuracy.
• Develop a scalable, model-agnostic framework that enhances SOTA MSA models without requiring retraining.
• Mitigate noise in generated contextual knowledge to ensure reliable fusion with input modalities.
• Demonstrate consistent performance gains across multiple benchmarks and model architectures.

Proposed method

• Use ChatGPT to generate prompt templates that guide LVLMs to generate contextually relevant world knowledge from image-text pairs.
• Employ a large vision-language model (LVLM) to generate contextual world knowledge (referred to as 'context') based on input images and sentences.
• Introduce a training-free Contextual Fusion mechanism that selectively integrates context into the MSA model, reducing noise impact.
• Apply the framework as a plug-in to existing MSA models, enabling universal enhancement across different architectures (e.g., LLaVA, AoM, mPLUG-Owl2).
• Use incremental context injection to analyze the contribution of individual context sentences, validating the importance of historical and event-based context.
• Visualize error rates on hard samples (low confidence, δ ≤ 0.3) to evaluate the effectiveness of Contextual Fusion in reducing mispredictions.

Experimental results

Research questions

• RQ1 Which types of world knowledge (e.g., historical, political, cultural) are most beneficial for improving multimodal sentiment analysis?
• RQ2 How does the inclusion of contextual world knowledge affect sentiment prediction performance, especially for ambiguous or hard samples?
• RQ3 To what extent does the Contextual Fusion mechanism reduce noise from generated context and improve robustness on challenging predictions?
• RQ4 Does the performance gain from WisdoM scale with LVLM size and training data volume?
• RQ5 Can the framework be universally applied to various SOTA MSA models without retraining?

Key findings

• WisdoM achieves an average F1 improvement of +1.89 across five advanced MSA methods, with gains up to +5.8 F1 on hard samples.
• Incorporating historical context significantly improves predictions—e.g., correctly identifying negative sentiment for 'Aleppo' by linking it to the Syrian war, which baseline models miss.
• The Contextual Fusion mechanism reduces error rates on hard samples (δ ≤ 0.3) compared to baseline models like AoM and LLaVA-v1.5, demonstrating robustness.
• Performance improves with larger LVLMs, and gains are pronounced even at 25% of the training data, indicating strong generalization and scalability.
• Historical and political world knowledge consistently yield the highest improvements across datasets, outperforming other types like artistic or literary context.
• Incremental context injection shows that adding event consequences leads to a notable 18% increase in negative sentiment probability, confirming the value of contextual depth.