[Paper Review] Overview of the CLEF-2021 CheckThat! Lab Task 2 on detecting previously fact-checked claims in tweets and political debates

Shaden Shaar, Fatima Haouari+7 more|Qatar University QSpace (Qatar University)|Jan 1, 2021

Misinformation and Its Impacts28 references18 citations

TL;DR

This paper presents the 2021 CLEF-2021 CheckThat! Lab, a multilingual fact-checking evaluation featuring three tasks: identifying check-worthy claims in tweets (Task 1), retrieving previously fact-checked claims for verification (Task 2), and detecting fake news and classifying topical domains in news articles (Task 3). The study demonstrates strong performance using transformer-based models, with the best system achieving 0.881 macro-F1 in claim verification and 0.905 accuracy in domain classification.

ABSTRACT

We describe the fourth edition of the CheckThat! Lab, part of the 2021 Conference and Labs of the Evaluation Forum (CLEF). The lab evaluates technology supporting three tasks related to factuality, and it covers Arabic, Bulgarian, English, Spanish, and Turkish. Here, we present the task 2, which asks to detect previously fact-checked claims (in two languages). A total of four teams participated in this task, submitted a total of sixteen runs, and most submissions managed to achieve sizable improvements over the baselines using transformer based models such as BERT, RoBERTa. In this paper, we describe the process of data collection and the task setup, including the evaluation measures used, and we give a brief overview of the participating systems. Last but not least, we release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in detecting previously fact-checked claims.

Motivation & Objective

• To develop and evaluate automated systems for detecting claims that are worth fact-checking in multilingual social media and political discourse.
• To enable retrieval of previously fact-checked claims to prevent redundant fact-checking efforts.
• To classify the veracity and topical domain of news articles to support expert routing and automated verification.
• To advance multilingual fact-checking technology by including Arabic, Bulgarian, English, Spanish, and Turkish.
• To support human fact-checkers through a pipeline of check-worthiness estimation, claim retrieval, and fake news detection.

Proposed method

• Task 1 uses ranking models to predict check-worthiness of tweets on COVID-19 and politics using BERT, RoBERTa, and AraBERT for multilingual input.
• Task 2 employs fine-tuned transformer models (e.g., RoBERTa) to rank previously fact-checked claims based on semantic similarity to a target claim.
• Task 3 applies multi-class classification with transformer architectures (e.g., BERT, ensemble models) to predict both veracity and topical domain of news articles.
• Evaluation metrics include mean average precision (MAP) for ranking tasks and macro-F1 for classification tasks.
• The system pipeline integrates claim detection, evidence retrieval, and veracity prediction to support semi-automatic fact-checking workflows.
• Datasets were constructed from real-world sources: Twitter streams, political debates, and news articles in five languages.

Experimental results

Research questions

• RQ1 How effective are transformer-based models in identifying check-worthy claims in multilingual social media content?
• RQ2 To what extent can previously fact-checked claims be retrieved using semantic matching for claim verification?
• RQ3 Can multi-class fake news detection models accurately predict both veracity and topical domain of news articles?
• RQ4 How does performance vary across languages (Arabic, English, Bulgarian, Spanish, Turkish) in fact-checking tasks?
• RQ5 What is the impact of ensemble models and fine-tuned transformers on the accuracy of claim verification and domain classification?

Key findings

• The best-performing system for Task 2 (previously fact-checked claim retrieval) achieved a macro-F1 score of 0.881 using an ensemble of three transformer models.
• For Task 3B (topical domain classification), the top team (NITK_NLP) achieved a macro-F1 of 0.881 and an accuracy of 0.905, with high per-class F1 scores (e.g., 0.950 for Climate, 0.946 for Health).
• The second-best system for Task 3B (NoFake) achieved a macro-F1 of 0.855, indicating strong performance across diverse domains including Crime (0.875) and Education (0.957).
• Task 1 saw strong performance with BERT and RoBERTa models, particularly in English and Arabic, with top systems using AraBERT and multilingual BERT.
• Task 3A (fake news detection) showed lower overall performance, with the best system achieving a macro-F1 of 0.841, indicating challenges in distinguishing fake from real news.
• The lab attracted 132 registered teams, with 15, 5, and 25 official submissions for Tasks 1, 2, and 3 respectively, making it the most popular CLEF-2021 lab.