[Paper Review] Breaking Language Barriers with MMTweets: Advancing Cross-Lingual Debunked Narrative Retrieval for Fact-Checking

Iknoor Singh, Carolina Scarton+2 more|arXiv (Cornell University)|Aug 10, 2023

Misinformation and Its ImpactsSocial Sciences3 citations

TL;DR

This paper introduces MMTweets, a multilingual dataset of 1,600 misinformation tweets in Hindi, Spanish, Portuguese, and English paired with cross-lingual fact-checking articles, and proposes a multistage retrieval framework that outperforms BM25 and fine-tuned multilingual Transformers in cross-lingual debunked narrative retrieval, enabling zero-shot and cross-domain transfer without task-specific fine-tuning.

ABSTRACT

Finding previously debunked narratives involves identifying claims that have already undergone fact-checking. The issue intensifies when similar false claims persist in multiple languages, despite the availability of debunks for several months in another language. Hence, automatically finding debunks (or fact-checks) in multiple languages is crucial to make the best use of scarce fact-checkers' resources. Mainly due to the lack of readily available data, this is an understudied problem, particularly when considering the cross-lingual scenario, i.e. the retrieval of debunks in a language different from the language of the online post being checked. This study introduces cross-lingual debunked narrative retrieval and addresses this research gap by: (i) creating Multilingual Misinformation Tweets (MMTweets): a dataset that stands out, featuring cross-lingual pairs, images, human annotations, and fine-grained labels, making it a comprehensive resource compared to its counterparts; (ii) conducting an extensive experiment to benchmark state-of-the-art cross-lingual retrieval models and introducing multistage retrieval methods tailored for the task; and (iii) comprehensively evaluating retrieval models for their cross-lingual and cross-dataset transfer capabilities within MMTweets, and conducting a retrieval latency analysis. We find that MMTweets presents challenges for cross-lingual debunked narrative retrieval, highlighting areas for improvement in retrieval models. Nonetheless, the study provides valuable insights for creating MMTweets datasets and optimising debunked narrative retrieval models to empower fact-checking endeavours. The dataset and annotation codebook are publicly available at https://doi.org/10.5281/zenodo.10637161.

Motivation & Objective

• Address the challenge of retrieving already debunked narratives in a language different from the query tweet, particularly in low-resource multilingual settings.
• Overcome the lack of annotated datasets for cross-lingual fact-checking retrieval, especially for non-English languages.
• Develop a robust retrieval system that generalizes across languages and domains without requiring task-specific fine-tuning.
• Enable zero-shot and cross-dataset transfer learning for fact-checking systems to reduce reliance on large-scale annotated data.
• Facilitate automated fact-checking by retrieving previously debunked claims across languages, reducing manual workload and misinformation spread.

Proposed method

• Construct the MMTweets dataset with 1,600 multilingual tweets and 30,452 fact-checking articles, annotated for cross-lingual relevance.
• Implement a multistage retrieval framework that separates the process into refinement and re-ranking stages to improve cross-lingual alignment.
• Use off-the-shelf multilingual pre-trained Transformer (MPT) models (e.g., mBERT, XLM-R) for encoding queries and documents in multiple languages.
• Apply lexical retrieval with BM25 as a strong baseline, using machine translation to align non-English queries with English fact-checks.
• Cache sentence embeddings in the multistage framework to optimize inference speed and enable efficient retrieval.
• Evaluate the framework on both MMTweets and CLEF Arabic datasets using cross-lingual and cross-dataset settings to test zero-shot generalization.

Experimental results

Research questions

• RQ1 Can off-the-shelf multilingual pre-trained models outperform lexical baselines like BM25 in cross-lingual debunked narrative retrieval without fine-tuning?
• RQ2 How effective is the proposed multistage retrieval framework in improving retrieval performance across multiple languages and datasets?
• RQ3 To what extent can the multistage framework generalize to zero-shot and cross-dataset scenarios, especially for low-resource languages?
• RQ4 What are the main failure modes of retrieval models when misinformation spans multiple modalities (e.g., text and image)?
• RQ5 Can the multistage framework enable effective knowledge transfer between datasets and languages, reducing the need for extensive fine-tuning?

Key findings

• Off-the-shelf multilingual pre-trained models fail to outperform the BM25 baseline when retrieving cross-lingual debunked narratives using machine-translated queries.
• The proposed multistage retrieval framework consistently outperforms both BM25 and fine-tuned MPT models across all languages tested, including Hindi, Spanish, Portuguese, and English.
• The multistage framework enables effective zero-shot and cross-dataset generalization, achieving strong performance on unseen languages and datasets without fine-tuning.
• The framework reduces reliance on expensive machine translation by caching embeddings, improving inference efficiency despite higher computational cost than BM25.
• Error analysis reveals that retrieval failures often occur when a single tweet is associated with multiple fact-checking articles, causing score dilution across relevant results.
• Multimodal misinformation—where text alone is insufficient due to misleading images or videos—poses a significant challenge, indicating the need for future multimodal retrieval systems.