[Paper Review] A Multi-Modal Method for Satire Detection using Textual and Visual Cues

Lily Li, Or Levi+2 more|arXiv (Cornell University)|Oct 13, 2020

Humor Studies and Applications13 references10 citations

TL;DR

This paper proposes a multi-modal satire detection model combining ViLBERT for joint text-image understanding and an image forensics technique to detect manipulation, outperforming unimodal and simple fusion baselines on a newly created dataset of 10,000 news headlines and thumbnails. The approach achieves 98.03% F1 score, demonstrating that joint visual-textual reasoning significantly improves satire detection over text-only or image-only methods.

ABSTRACT

Satire is a form of humorous critique, but it is sometimes misinterpreted by readers as legitimate news, which can lead to harmful consequences. We observe that the images used in satirical news articles often contain absurd or ridiculous content and that image manipulation is used to create fictional scenarios. While previous work have studied text-based methods, in this work we propose a multi-modal approach based on state-of-the-art visiolinguistic model ViLBERT. To this end, we create a new dataset consisting of images and headlines of regular and satirical news for the task of satire detection. We fine-tune ViLBERT on the dataset and train a convolutional neural network that uses an image forensics technique. Evaluation on the dataset shows that our proposed multi-modal approach outperforms image-only, text-only, and simple fusion baselines.

Motivation & Objective

• To address the growing problem of satirical news being mistaken for real news, which can spread misinformation.
• To investigate whether visual cues—especially image manipulation—can improve satire detection beyond text-based methods.
• To create a new multi-modal dataset of 10,000 news articles (6,000 factual, 4,000 satirical) with headlines and thumbnails for satire detection research.
• To evaluate the effectiveness of multi-modal fusion, particularly using a pre-trained visiolinguistic model like ViLBERT, in distinguishing satire from real news.

Proposed method

• The authors create a new dataset of 10,000 news articles (6,000 factual, 4,000 satirical) from 10 reputable sources, including mainstream outlets and known satirical websites.
• They fine-tune the Vision-and-Language BERT (ViLBERT) model on the dataset, leveraging its co-attentive transformer architecture to jointly process text and image features.
• An additional model, ELA+CNN, is trained to detect image manipulations using Edge-Enhanced Laplacian of Gaussian (ELA) preprocessing and a convolutional neural network.
• The performance of ViLBERT is compared against unimodal baselines (BERT BASE for text, ResNet-101 and ELA+CNN for images) and simple fusion methods (average and concatenation fusion).
• The model uses early, deep fusion via co-attentive layers that allow visual and textual representations to interact during attention computation.
• The dataset is split into training, validation, and test sets, with evaluation using accuracy, F1 score, and AUC-ROC metrics.

Experimental results

Research questions

• RQ1 Can visual cues, particularly image manipulation, improve the detection of satirical news beyond textual analysis alone?
• RQ2 Does a multi-modal approach using a pre-trained visiolinguistic model like ViLBERT outperform unimodal or simple fusion methods in satire detection?
• RQ3 To what extent do image forensics techniques such as ELA detect manipulations in satirical news thumbnails?
• RQ4 Why do some satirical or factual news articles get misclassified by the model, and what are the primary failure modes?

Key findings

• ViLBERT achieved the highest performance with a 98.03% F1 score, significantly outperforming the next best model, BERT BASE, which achieved 93.80% F1.
• The image-only model ELA+CNN performed worse than random chance, achieving only 44.20% accuracy, likely due to image resaving and compression reducing detectable error patterns.
• The simple fusion models (average and concatenation) showed only marginal improvement over BERT BASE, indicating that early, deep fusion is more effective than late or element-wise fusion.
• Misclassification analysis revealed three main failure modes: misinterpreting metaphorical language (e.g., 'bursts' in headlines), lacking political knowledge (e.g., not recognizing absurd political contrasts), and failing to distinguish between bizarre but true stories and fabricated content.
• The model struggled with headlines using figurative language, even when images were unaltered, suggesting limitations in understanding linguistic nuance.
• Despite the poor performance of ELA+CNN, the study confirms that image forensics can be a valuable signal when combined with deep multi-modal learning, especially when the images are not heavily compressed.