[Paper Review] Multi-modal Fusion using Fine-tuned Self-attention and Transfer Learning for Veracity Analysis of Web Information

Priyanka Meel, Dinesh Kumar Vishwakarma|arXiv (Cornell University)|Sep 26, 2021

Misinformation and Its Impacts4 citations

TL;DR

This paper proposes a multi-modal veracity analysis framework that fuses textual and visual features using fine-tuned BERT/ALBERT and Inception-ResNet-v2 models with early and late fusion strategies. It achieves state-of-the-art performance on three diverse datasets—All Data, Weibo, and MediaEval 2016—demonstrating the effectiveness of transfer learning and self-attention in detecting fake news across languages and modalities.

ABSTRACT

The nuisance of misinformation and fake news has escalated many folds since the advent of online social networks. Human consciousness and decision-making capabilities are negatively influenced by manipulated, fabricated, biased or unverified news posts. Therefore, there is a high demand for designing veracity analysis systems to detect fake information contents in multiple data modalities. In an attempt to find a sophisticated solution to this critical issue, we proposed an architecture to consider both the textual and visual attributes of the data. After the data pre-processing is done, text and image features are extracted from the training data using separate deep learning models. Feature extraction from text is done using BERT and ALBERT language models that leverage the benefits of bidirectional training of transformers using a deep self-attention mechanism. The Inception-ResNet-v2 deep neural network model is employed for image data to perform the task. The proposed framework focused on two independent multi-modal fusion architectures of BERT and Inception-ResNet-v2 as well as ALBERT and Inception-ResNet-v2. Multi-modal fusion of textual and visual branches is extensively experimented and analysed using concatenation of feature vectors and weighted averaging of probabilities named as Early Fusion and Late Fusion respectively. Three publicly available broadly accepted datasets All Data, Weibo and MediaEval 2016 that incorporates English news articles, Chinese news articles, and Tweets correspondingly are used so that our designed framework's outcomes can be properly tested and compared with previous notable work in the domain.

Motivation & Objective

• To address the growing challenge of misinformation and fake news on online social networks.
• To design a robust veracity analysis system capable of processing both textual and visual content from diverse web sources.
• To evaluate the effectiveness of multi-modal fusion strategies—early and late fusion—on cross-lingual and multi-modal datasets.
• To leverage transfer learning and self-attention mechanisms to improve feature representation for fake news detection.

Proposed method

• Textual features are extracted using fine-tuned BERT and ALBERT models, leveraging their deep self-attention mechanisms for contextual representation.
• Visual features are extracted using the Inception-ResNet-v2 model pre-trained on ImageNet for robust image representation.
• Early fusion combines the final feature vectors of text and image branches through concatenation before classification.
• Late fusion combines the predicted probabilities from separate text and image classifiers using weighted averaging.
• The framework is evaluated on three public datasets: All Data (English), Weibo (Chinese), and MediaEval 2016 (Tweets) to ensure cross-lingual generalization.
• Transfer learning is applied to both text and image encoders to improve performance on low-resource and domain-specific data.

Experimental results

Research questions

• RQ1 How effective is multi-modal fusion using early and late fusion strategies in detecting fake news across different data modalities?
• RQ2 Can fine-tuned BERT and ALBERT models improve veracity detection when combined with visual features from deep convolutional networks?
• RQ3 How does the proposed framework perform across diverse, real-world datasets with varying languages and content types?
• RQ4 What is the relative contribution of textual versus visual features in detecting misinformation in web content?
• RQ5 Does transfer learning enhance the performance of multi-modal veracity analysis systems on low-resource or noisy social media data?

Key findings

• The proposed framework achieves state-of-the-art performance on the All Data, Weibo, and MediaEval 2016 datasets, outperforming previous methods in multi-modal fake news detection.
• Late fusion with weighted averaging of probabilities consistently yields higher accuracy than early fusion across all three datasets.
• Fine-tuned BERT and ALBERT models significantly improve textual representation, especially in capturing nuanced linguistic cues in misinformation.
• Inception-ResNet-v2 provides robust visual feature extraction, enabling effective detection even when textual content is minimal or misleading.
• The integration of transfer learning and self-attention mechanisms leads to improved generalization across languages and data types.
• The model demonstrates strong performance on low-resource and multilingual settings, particularly on the Chinese Weibo dataset.