[Paper Review] SAFE: Similarity-Aware Multi-Modal Fake News Detection

Xinyi Zhou, Jindi Wu+1 more|arXiv (Cornell University)|Feb 19, 2020

Misinformation and Its ImpactsSocial Sciences29 references94 citations

TL;DR

SAFE jointly models textual and visual content with their cross-modal similarity to detect fake news, outperforming baselines on PolitiFact and GossipCop datasets.

ABSTRACT

Effective detection of fake news has recently attracted significant attention. Current studies have made significant contributions to predicting fake news with less focus on exploiting the relationship (similarity) between the textual and visual information in news articles. Attaching importance to such similarity helps identify fake news stories that, for example, attempt to use irrelevant images to attract readers' attention. In this work, we propose a $\mathsf{S}$imilarity-$\mathsf{A}$ware $\mathsf{F}$ak$\mathsf{E}$ news detection method ($\mathsf{SAFE}$) which investigates multi-modal (textual and visual) information of news articles. First, neural networks are adopted to separately extract textual and visual features for news representation. We further investigate the relationship between the extracted features across modalities. Such representations of news textual and visual information along with their relationship are jointly learned and used to predict fake news. The proposed method facilitates recognizing the falsity of news articles based on their text, images, or their "mismatches." We conduct extensive experiments on large-scale real-world data, which demonstrate the effectiveness of the proposed method.

Motivation & Objective

• Motivation to improve fake news detection by leveraging cross-modal similarity between text and images.
• Develop a multi-modal framework that learns textual, visual, and relational representations of news articles.
• Demonstrate that incorporating cross-modal similarity improves fake news prediction over single-modal baselines.

Proposed method

• Extract textual features using an extended Text-CNN with an extra fully connected layer.
• Extract visual features via a Text-CNN on features from a pre-trained image2sentence pipeline, followed by a transform to a d-dimensional representation.
• Define a cross-modal similarity measure s between text and image representations and a dedicated loss term for similarity-based signaling of mismatch.
• Combine multi-modal features and cross-modal similarity through a joint learning objective with two loss components (prediction loss and similarity loss).
• Optimize end-to-end with a joint gradient-based procedure (Algorithm 1) to update textual, visual, and predictor parameters.

Experimental results

Research questions

• RQ1 Can a similarity-aware multi-modal model detect fake news more effectively than single-modal or naive multi-modal baselines?
• RQ2 Does incorporating cross-modal similarity help identify mismatches between text and images that indicate fakery?
• RQ3 What is the contribution balance between multi-modal content and cross-modal similarity for accurate fake news detection?

Key findings

• SAFE achieves higher accuracy and F1 scores than LIWC, VGG-19, and att-RNN baselines on PolitiFact and GossipCop datasets.
• Integrating textual, visual, and cross-modal similarity outperforms variants that omit either modality or the similarity term.
• Textual information tends to be more informative than visual information, but both modalities plus their relationship yield the best results.
• The method remains effective when adjusting the relative weights between multi-modal features and cross-modal similarity, indicating both contribute meaningfully to detection.
• Case studies show lower similarity scores often align with fake news, illustrating the model’s capability to flag mismatches.