[Paper Review] Automated Fact Checking: Task formulations, methods and future directions

James Thorne, Andreas Vlachos|arXiv (Cornell University)|Jun 20, 2018

Misinformation and Its Impacts29 references130 citations

TL;DR

This survey unifies task formulations and methodologies across NLP and related fields for automated fact checking, emphasizing evidence as a key distinguishing factor and outlining future NLP directions.

ABSTRACT

The recently increased focus on misinformation has stimulated research in fact checking, the task of assessing the truthfulness of a claim. Research in automating this task has been conducted in a variety of disciplines including natural language processing, machine learning, knowledge representation, databases, and journalism. While there has been substantial progress, relevant papers and articles have been published in research communities that are often unaware of each other and use inconsistent terminology, thus impeding understanding and further progress. In this paper we survey automated fact checking research stemming from natural language processing and related disciplines, unifying the task formulations and methodologies across papers and authors. Furthermore, we highlight the use of evidence as an important distinguishing factor among them cutting across task formulations and methods. We conclude with proposing avenues for future NLP research on automated fact checking.

Motivation & Objective

• Clarify and unify definitions and task formulations across NLP, ML, knowledge representation and journalism for automated fact checking.
• Examine the role and types of evidence used in fact checking and how they influence inputs and outputs.
• Review datasets, models, and evaluation paradigms to identify gaps and future research directions.

Proposed method

• Classify inputs into textual claims, triples, or documents and discuss grounding/disambiguation needs.
• Survey evidence sources including knowledge graphs, textual sources, and prior fact-checked repositories.
• Compare output formulations from binary, ordinal, to multi-label and score-based verdicts with corresponding evaluation signals.
• Discuss supervised learning as the dominant approach and how evidence retrieval and entailment/ranking methods are integrated.
• Highlight pipeline architectures such as document retrieval, sentence selection, and textual entailment models in FEVER-like setups.

Experimental results

Research questions

• RQ1 What are the common inputs and outputs used across automated fact-checking research?
• RQ2 What evidence types are employed and how do they affect model design and evaluation?
• RQ3 How do datasets shape the development and evaluation of fact-checking models?
• RQ4 What are the primary methodological approaches (e.g., textual entailment, knowledge graphs) and their limitations?
• RQ5 What future NLP directions are promising for scalable and trustworthy automated fact checking?

Key findings

• Evidence-grounded approaches are central across task formulations, with knowledge graphs, text sources, and prior fact-checked claims serving as key inputs.
• Diverse outputs range from binary labels to multi-class and score-based verdicts, with FEVER-like tasks requiring evidence sentences alongside verdicts.
• Datasets vary in size and evidence availability, influencing the feasibility of machine learning approaches for fact checking.
• Supervised learning dominates current methods, often augmented with retrieval, grounding, and narrative or provenance considerations.
• Ethical and grounding challenges arise from reliance on originator profiles, credibility signals, and limitations of knowledge bases.