[Paper Review] Generating Fact Checking Briefs

Angela Fan, Aleksandra Piktus+6 more|arXiv (Cornell University)|Jan 1, 2020

Topic Modeling43 references4 citations

TL;DR

This paper introduces fact checking briefs—particularly Question-Answering Briefs (QABriefs)—to improve the accuracy and efficiency of human fact checking. Using a novel model, QABRIEFER, and a crowdsourced dataset (QABRIEFDATASET), the authors show that QABriefs increase crowdworker accuracy by 10% and reduce checking time by 20% compared to using only a search bar.

ABSTRACT

Fact checking at scale is difficult -- while the number of active fact checking websites is growing, it remains too small for the needs of the contemporary media ecosystem. However, despite good intentions, contributions from volunteers are often error-prone, and thus in practice restricted to claim detection. We investigate how to increase the accuracy and efficiency of fact checking by providing information about the claim before performing the check, in the form of natural language briefs. We investigate passage-based briefs, containing a relevant passage from Wikipedia, entity-centric ones consisting of Wikipedia pages of mentioned entities, and Question-Answering Briefs, with questions decomposing the claim, and their answers. To produce QABriefs, we develop QABriefer, a model that generates a set of questions conditioned on the claim, searches the web for evidence, and generates answers. To train its components, we introduce QABriefDataset which we collected via crowdsourcing. We show that fact checking with briefs -- in particular QABriefs -- increases the accuracy of crowdworkers by 10% while slightly decreasing the time taken. For volunteer (unpaid) fact checkers, QABriefs slightly increase accuracy and reduce the time required by around 20%.

Motivation & Objective

• Address the scalability and accuracy limitations of crowdsourced fact checking in the face of widespread misinformation.
• Improve human fact-checking performance by providing pre-checked, structured information before the verification process begins.
• Develop a method to automatically generate high-quality, claim-specific briefs that support fact-checking decisions.
• Create a large-scale, high-quality dataset of QABriefs to enable training and evaluation of fact-checking models.
• Demonstrate that briefs can reduce reliance on manual search while improving both speed and correctness in fact-checking workflows.

Proposed method

• Develop QABRIEFER, a two-stage model that first generates questions conditioned on a claim, then answers them using open-domain QA.
• Use dense passage retrieval (DPR) to extract relevant Wikipedia passages for passage-based briefs.
• Construct entity-centric briefs by linking mentioned entities to their Wikipedia pages.
• Leverage a large-scale, crowdsourced dataset (QABRIEFDATASET) with ~10,000 QABriefs, each containing ~3 question-answer pairs.
• Fine-tune a pre-trained sequence-to-sequence model (BART) on both Natural Questions (NQ) and QABRIEFDATASET for optimal QA performance.
• Integrate retrieval and generation components to produce end-to-end briefs that simulate expert-level background preparation.

Experimental results

Research questions

• RQ1 Can generating structured briefs before fact checking improve the accuracy and efficiency of crowdworkers?
• RQ2 How do different types of briefs—passage-based, entity-centric, and QABriefs—compare in supporting fact-checking performance?
• RQ3 To what extent can an automated model (QABRIEFER) generate QABriefs that match the quality of human-annotated briefs?
• RQ4 Does the use of briefs reduce over-reliance on general web search while maintaining or improving verification quality?
• RQ5 What impact do briefs have on fact-checker confidence and decision-making behavior?

Key findings

• QABriefs generated by QABRIEFER improved crowdworker accuracy by 10% compared to using only a search bar.
• Volunteer fact checkers using QABriefs achieved a 4% accuracy gain and completed checks 20% faster than with search-only access.
• Human-annotated QABriefs led to the highest performance gains, followed by QABRIEFER-generated briefs and other brief types.
• Providing QABriefs increased overconfidence: approximately 45% of crowdworkers did not use the search bar, even though search access improved accuracy.
• Fine-tuning BART on both Natural Questions (NQ) and QABRIEFDATASET yielded the best QA performance, highlighting the value of pre-training on large-scale QA data.
• The study demonstrates that briefs can support a more structured fact-checking process, especially when derived from professional fact-checking articles.