[Paper Review] Self-Supervised Claim Identification for Automated Fact Checking

Archita Pathak, Mohammad Abuzar Shaikh+1 more|arXiv (Cornell University)|Feb 3, 2021

Misinformation and Its Impacts32 references4 citations

TL;DR

This paper proposes a self-supervised, attention-based model (MA-CIN) that identifies 'claim-worthy' sentences in fake news articles by leveraging headline-content aboutness, eliminating the need for costly human annotations. The model achieves strong performance across three datasets and introduces a new benchmark dataset with veracity labels and evidence for automated fact-checking.

ABSTRACT

We propose a novel, attention-based self-supervised approach to identify "claim-worthy" sentences in a fake news article, an important first step in automated fact-checking. We leverage "aboutness" of headline and content using attention mechanism for this task. The identified claims can be used for downstream task of claim verification for which we are releasing a benchmark dataset of manually selected compelling articles with veracity labels and associated evidence. This work goes beyond stylistic analysis to identifying content that influences reader belief. Experiments with three datasets show the strength of our model. Data and code available at https://github.com/architapathak/Self-Supervised-ClaimIdentification

Motivation & Objective

• Address the lack of scalable, low-resource methods for identifying claim-worthy sentences in fake news articles.
• Overcome limitations of existing supervised and weakly supervised approaches that rely on expensive human annotations.
• Develop a self-supervised framework that leverages the semantic relationship (aboutness) between headlines and article content to detect influential, verifiable claims.
• Create a new benchmark dataset with manually labeled veracity labels and supporting evidence for fake news articles to support downstream fact-checking tasks.
• Enable scalable, generalizable claim detection that captures content influencing reader belief beyond stylistic cues.

Proposed method

• Propose a self-supervised, multi-head attention-based neural network (MA-CIN) that models the aboutness relationship between the article headline (query) and each sentence (document).
• Use the attention weights between the headline and each sentence as a proxy for claim-worthiness, where higher attention scores indicate greater relevance to the article's core claim.
• Train the model end-to-end using contrastive learning objectives that maximize attention to relevant sentences while minimizing attention to irrelevant ones, without requiring labeled claims.
• Leverage pre-trained sentence encoders (e.g., BERT) to initialize sentence and headline representations before attention computation.
• Apply a contrastive loss function to encourage high attention to sentences that are semantically aligned with the headline, based on the concept of 'aboutness' from information retrieval.
• Fine-tune the model on a small set of manually curated fake news articles with veracity labels and evidence to improve generalization.

Experimental results

Research questions

• RQ1 Can a self-supervised model effectively identify claim-worthy sentences in fake news without requiring human-annotated claims?
• RQ2 How well does the attention mechanism capture the semantic 'aboutness' between headlines and content compared to traditional supervised or weakly supervised methods?
• RQ3 To what extent does the proposed model generalize across diverse fake news datasets with varying topics and structures?
• RQ4 How does the model’s performance compare to supervised baselines when no labeled claims are available during training?
• RQ5 Can the model detect claims that are semantically relevant to the headline even when lexical overlap is minimal?

Key findings

• The MA-CIN model achieves strong performance on three distinct datasets—DNF_Eval, CDC_Eval, and a new benchmark—without requiring any human-annotated claims during training.
• In qualitative analysis, the model correctly identifies 3 out of the top 5 human-evaluated claims in a sample article, demonstrating strong alignment with human judgment on semantic relevance.
• The model successfully captures semantic aboutness: it assigns low attention to sentences with mismatched meaning (e.g., sentence 1, which shares words but not meaning with the headline) and high attention to semantically coherent claims (e.g., sentences 2, 12, 13).
• Attention patterns reveal that the model learns to group related sentences (e.g., sentences 3–8 form a narrative chain), and avoids self-attention on the same sentence, indicating robust reasoning.
• The model identifies some sentences (e.g., sentence 3 and 4) as relevant even when not in the ground truth, suggesting potential for broader claim detection beyond annotator consensus.
• The model’s performance is robust to subjective biases in human-annotated data, as shown by consistent attention patterns across multiple evaluation cases.