[Paper Review] Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution

Ron Zhu|arXiv (Cornell University)|Dec 15, 2020

Hate Speech and Cyberbullying Detection15 references46 citations

TL;DR

This work ensembles four vision-language transformers and augments them with external labels (entity/race/gender) and in-domain pretraining to win the Hateful Meme Detection Challenge, achieving 0.845 AUROC on the dataset.

ABSTRACT

Hateful meme detection is a new research area recently brought out that requires both visual, linguistic understanding of the meme and some background knowledge to performing well on the task. This technical report summarises the first place solution of the Hateful Meme Detection Challenge 2020, which extending state-of-the-art visual-linguistic transformers to tackle this problem. At the end of the report, we also point out the shortcomings and possible directions for improving the current methodology.

Motivation & Objective

• Motivate improved hateful meme detection through deeper multimodal understanding beyond image-text pairs.
• Incorporate external knowledge cues (entity, race, gender) to augment visual-linguistic representations.
• Leverage in-domain pretraining and ensemble methods to boost performance on the Hateful Memes dataset.
• Evaluate how feeding external labels interacts with different VL transformer architectures.

Proposed method

• Extend VL-BERT, UNITER, VILLA, and ERNIE-Vil into a four-model ensemble for hateful meme detection.
• Represent external labels as special text tokens linked to image regions to fuse with the multimodal backbone.
• Reuse UNITER's ITM head to provide image-text matching signals tuned for the task.
• Extract additional information from data sources via Google Vision Web Entity Detection and FairFace labels.
• Discuss the potential of knowledge graph–driven approaches (e.g., MHGRN) as future work for incorporating external knowledge.

Experimental results

Research questions

• RQ1 Can external labels and in-domain pretraining improve hateful meme detection beyond standard VL transformers?
• RQ2 How do external knowledge cues interact with different VL architectures in a multimodal hate-speech task?
• RQ3 Does reusing pre-trained heads (like UNITER ITM) reduce linguistic bias and improve generalization?
• RQ4 What is the contribution of additional data sources (web entities, demographic attributes) to model performance?

Key findings

• Achieved 0.845 AUROC on the hateful memes dataset with the proposed ensemble and extensions.
• Applying entity/race/gender tags significantly boosted dev-seen performance for extended VL-BERT and ERNIE-Vil (less so for some single-stream models).
• Reuse of UNITER ITM head provided advantages over random initialization (0.778 vs 0.765 AUROC on dev-seen).
• ERNIe-Vil and VL-BERT benefited from external labels due to cross-modal fusion advantages; results varied across architectures.
• Highlighting the importance of combining image, text, and external knowledge for meme polarity and the potential of knowledge-graph–based approaches for future work.