[Paper Review] Exploiting Multi-domain Visual Information for Fake News Detection

Peng Qi, Juan Cao+3 more|arXiv (Cornell University)|Aug 13, 2019

Misinformation and Its Impacts33 references33 citations

TL;DR

MVNN fuses frequency-domain CNN with a multi-branch pixel-domain CNN-RNN and attention to detect fake-news images, achieving 84.6% accuracy on a real-world dataset, outperforming baselines.

ABSTRACT

The increasing popularity of social media promotes the proliferation of fake news. With the development of multimedia technology, fake news attempts to utilize multimedia contents with images or videos to attract and mislead readers for rapid dissemination, which makes visual contents an important part of fake news. Fake-news images, images attached in fake news posts,include not only fake images which are maliciously tampered but also real images which are wrongly used to represent irrelevant events. Hence, how to fully exploit the inherent characteristics of fake-news images is an important but challenging problem for fake news detection. In the real world, fake-news images may have significantly different characteristics from real-news images at both physical and semantic levels, which can be clearly reflected in the frequency and pixel domain, respectively. Therefore, we propose a novel framework Multi-domain Visual Neural Network (MVNN) to fuse the visual information of frequency and pixel domains for detecting fake news. Specifically, we design a CNN-based network to automatically capture the complex patterns of fake-news images in the frequency domain; and utilize a multi-branch CNN-RNN model to extract visual features from different semantic levels in the pixel domain. An attention mechanism is utilized to fuse the feature representations of frequency and pixel domains dynamically. Extensive experiments conducted on a real-world dataset demonstrate that MVNN outperforms existing methods with at least 9.2% in accuracy, and can help improve the performance of multimodal fake news detection by over 5.2%.

Motivation & Objective

• Motivate the need to utilize visual content in fake news detection beyond text and social context.
• Propose a multi-domain visual framework (MVNN) to capture physical and semantic features of fake-news images.
• Demonstrate that fusing frequency-domain and pixel-domain features improves detection performance.
• Validate MVNN on a real-world dataset and compare against baselines.

Proposed method

• Transform images to frequency domain with block DCT and a CNN to capture physical tampering/recompression patterns.
• Use a multi-branch CNN-RNN (Bi-GRU) in the pixel domain to extract multi-level semantic features from the image.
• Fuse frequency and pixel features via an attention mechanism to produce a final fake-news vs real-news classification.
• Train end-to-end with cross-entropy loss and use dropout, batch normalization, and Adam optimization.
• Evaluate with accuracy, precision, recall, and F1 on the Weibo fake-news dataset.

Experimental results

Research questions

• RQ1 Can MVNN improve fake-news detection using visual modality alone?
• RQ2 How do frequency-domain and pixel-domain components, plus attention and architectural choices, affect MVNN's performance?
• RQ3 Can MVNN enhance multimodal fake-news detection when combined with text?
• RQ4 What is the contribution of each domain and network component to overall performance?

Key findings

• MVNN achieves 84.6% accuracy in single-visual-modality fake-news detection, outperforming baselines by at least 9.2%.
• Pixel-domain features contribute more than frequency-domain features, with frequency providing auxiliary benefit.
• Attention fusion, Bi-GRU, and multiple branches in the pixel-domain sub-network significantly improve performance.
• Ablation shows removing frequency/pixel domains reduces accuracy to 0.794/0.737 respectively, and removing attention, Bi-GRU, or branches also degrades results.
• Visual feature spaces learned by MVNN show clearer separation between fake and real images than either domain alone.