[Paper Review] MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets

Shraman Pramanick, Shivam Sharma+4 more|arXiv (Cornell University)|Sep 11, 2021

Hate Speech and Cyberbullying Detection4 citations

TL;DR

MOMENTA is a novel multimodal deep learning framework that detects harmful memes and identifies their social targets by analyzing both local and global image-text interactions in context. It outperforms ten baselines by 1.3–2.6% absolute accuracy on two new datasets—Harm-C (COVID-19) and Harm-P (U.S. politics)—and demonstrates strong interpretability and generalization across domains.

ABSTRACT

Internet memes have become powerful means to transmit political, psychological, and socio-cultural ideas. Although memes are typically humorous, recent days have witnessed an escalation of harmful memes used for trolling, cyberbullying, and abuse. Detecting such memes is challenging as they can be highly satirical and cryptic. Moreover, while previous work has focused on specific aspects of memes such as hate speech and propaganda, there has been little work on harm in general. Here, we aim to bridge this gap. We focus on two tasks: (i)detecting harmful memes, and (ii)identifying the social entities they target. We further extend a recently released HarMeme dataset, which covered COVID-19, with additional memes and a new topic: US politics. To solve these tasks, we propose MOMENTA (MultimOdal framework for detecting harmful MemEs aNd Their tArgets), a novel multimodal deep neural network that uses global and local perspectives to detect harmful memes. MOMENTA systematically analyzes the local and the global perspective of the input meme (in both modalities) and relates it to the background context. MOMENTA is interpretable and generalizable, and our experiments show that it outperforms several strong rivaling approaches.

Motivation & Objective

• To address the gap in detecting harmful memes that are not overtly hateful or offensive but still damaging due to context and multimodal synergy.
• To identify the specific social entities targeted by harmful memes, a task underexplored in prior work on hate or offensiveness detection.
• To extend the HarMeme dataset with new examples and a new topic (U.S. politics), enabling broader evaluation of harm detection systems.
• To develop a model that integrates background context effectively, improving detection beyond unimodal or early-fusion approaches.
• To ensure interpretability and generalizability of the model across diverse meme types and domains.

Proposed method

• MOMENTA employs a dual-branch architecture that separately encodes image and text features using pre-trained vision and language encoders (e.g., ViT and BERT).
• It applies local and global attention mechanisms to capture fine-grained visual cues (e.g., gestures, facial expressions) and global semantic coherence between image and text.
• The framework fuses multimodal representations with background context embeddings derived from external knowledge sources or contextual encoders.
• It uses a contrastive learning objective to align harmful meme representations with their targets, improving discrimination of subtle harm.
• Interpretability is achieved via Grad-CAM and LIME-based saliency maps, highlighting key image regions and text tokens contributing to predictions.
• The model is fine-tuned end-to-end on the extended HarMeme dataset, with adversarial training and normalization techniques to improve robustness.

Experimental results

Research questions

• RQ1 How does integrating background context improve the detection of subtle, context-dependent harmful memes?
• RQ2 To what extent can a multimodal model distinguish harmful memes that are not overtly offensive or hateful but still damaging?
• RQ3 Can a unified framework effectively detect both harmful memes and their targeted social entities in a single architecture?
• RQ4 How does MOMENTA’s performance compare to strong unimodal and multimodal baselines on diverse meme detection tasks?
• RQ5 What are the key visual and textual cues that contribute to the model’s predictions, and how interpretable are they?

Key findings

• MOMENTA achieves 1.3–2.6% absolute accuracy gain over ten strong baselines on both the Harm-C and Harm-P datasets for harmful meme detection.
• The model demonstrates superior generalization, performing well on unseen topics and domains, including U.S. politics beyond the original COVID-19 focus.
• Interpretability analysis reveals that key harmful cues—such as conflicting gestures and specific words like 'CAMPAIGNING'—are correctly highlighted by saliency maps.
• Despite high performance, the model fails on memes where harm arises from entrenched semantic meaning rather than explicit visual or textual cues, indicating a need for deeper contextual reasoning.
• The framework successfully identifies the target of a meme (e.g., Joe Biden, Donald Trump) with high precision, a capability not widely supported in prior work.
• The ablation study confirms that both local and global multimodal modeling, along with background context, are essential for optimal performance.