[Paper Review] Too Many Claims to Fact-Check: Prioritizing Political Claims Based on Check-Worthiness

Yavuz Selim Kartal, Busra Guvenen+1 more|arXiv (Cornell University)|Apr 17, 2020

Misinformation and Its Impacts22 references4 citations

TL;DR

This paper proposes a BERT-based hybrid model that prioritizes political claims for fact-checking by combining fine-tuned BERT with handcrafted features such as word embeddings, part-of-speech tags, comparative/superlative clauses, and domain-specific controversial topics. The model achieves state-of-the-art performance with 0.255 MAP on CLEF Check That! 2018 and 0.176 MAP on 2019, outperforming all prior methods.

ABSTRACT

The massive amount of misinformation spreading on the Internet on a daily basis has enormous negative impacts on societies. Therefore, we need automated systems helping fact-checkers in the combat against misinformation. In this paper, we propose a model prioritizing the claims based on their check-worthiness. We use BERT model with additional features including domain-specific controversial topics, word embeddings, and others. In our experiments, we show that our proposed model outperforms all state-of-the-art models in both test collections of CLEF Check That! Lab in 2018 and 2019. We also conduct a qualitative analysis to shed light-detecting check-worthy claims. We suggest requesting rationales behind judgments are needed to understand subjective nature of the task and problematic labels.

Motivation & Objective

• To address the challenge of prioritizing political claims for fact-checking due to the overwhelming volume of misinformation.
• To develop an automated system that ranks claims by their check-worthiness to help fact-checkers focus on the most critical claims.
• To improve performance over existing state-of-the-art models in claim prioritization tasks using hybrid deep learning and feature engineering.
• To investigate the subjective and inconsistent nature of check-worthiness annotations through qualitative analysis.
• To advocate for collecting rationales behind annotations to improve model interpretability and data quality.

Proposed method

• Fine-tune a BERT model on claim prioritization using the CLEF Check That! datasets from 2018 and 2019.
• Combine BERT’s contextual representations with additional features: word embeddings, POS tags, comparative/superlative clause indicators, and domain-specific controversial topics.
• Train a logistic regression classifier on the ensemble of BERT predictions and engineered features to rank claims by check-worthiness.
• Use feature ablation studies to evaluate the contribution of each component to model performance.
• Conduct qualitative analysis on model predictions and annotations to assess label subjectivity and inconsistencies.
• Request rationales for annotations to better understand human judgment variability and improve future data collection.

Experimental results

Research questions

• RQ1 How can we effectively prioritize political claims for fact-checking in the face of massive misinformation?
• RQ2 To what extent does combining BERT with handcrafted features improve check-worthiness ranking over standalone models?
• RQ3 How do subjective and inconsistent human annotations affect the reliability of claim prioritization benchmarks?
• RQ4 Can collecting rationales for annotations help resolve disagreements and improve model training and evaluation?
• RQ5 What features are most predictive of check-worthiness in political claims?

Key findings

• The proposed BERT-based hybrid model achieves a mean average precision (MAP) of 0.255 on the CLEF Check That! 2018 test collection, outperforming all prior state-of-the-art models.
• On the CLEF Check That! 2019 test collection, the model achieves a MAP of 0.176, again surpassing all previous methods.
• Feature ablation shows that BERT and word embeddings are the most influential features, while handcrafted word lists and domain-specific controversial topics also contribute significantly.
• Qualitative analysis reveals that check-worthiness annotations are highly subjective, with inconsistent labeling even for semantically similar claims.
• The study identifies problematic labels, such as labeling future-tense claims or non-predicate statements as check-worthy, suggesting annotation noise.
• The authors conclude that collecting rationales for annotations is essential to understand human judgment variability and improve future systems.