[Paper Review] Real-time Claim Detection from News Articles and Retrieval of Semantically-Similar Factchecks

Ben Adler, Giacomo Boscaini-Gilroy|arXiv (Cornell University)|Jul 3, 2019

Topic Modeling14 references6 citations

TL;DR

This paper proposes a real-time system for detecting claims in news articles and retrieving semantically similar, previously factchecked claims using contextual sentence embeddings and dynamic clustering. It leverages Google's Universal Sentence Encoder Large for high-accuracy claim embedding and combines DBScan with Louvain community detection to build compact, scalable clusters, achieving efficient, low-latency retrieval with ~300ms per claim.

ABSTRACT

Factchecking has always been a part of the journalistic process. However with newsroom budgets shrinking it is coming under increasing pressure just as the amount of false information circulating is on the rise. We therefore propose a method to increase the efficiency of the factchecking process, using the latest developments in Natural Language Processing (NLP). This method allows us to compare incoming claims to an existing corpus and return similar, factchecked, claims in a live system-allowing factcheckers to work simultaneously without duplicating their work.

Motivation & Objective

• To address the growing challenge of misinformation by automating and scaling the factchecking process in newsrooms under budget constraints.
• To detect claims in news articles automatically, reducing reliance on manual monitoring and duplication of effort.
• To retrieve semantically similar, previously factchecked claims to enable reuse of existing factchecking work.
• To develop a scalable, real-time system that clusters claims using contextual embeddings and dynamic clustering.
• To improve efficiency and reduce cost in factchecking by leveraging state-of-the-art NLP techniques.

Proposed method

• The system uses a combination of PolitiTax and Full Fact’s claim definitions to create a labeled dataset for claim detection.
• It employs Google’s Universal Sentence Encoder Large (USE Large) as the primary embedding model, outperforming TF-IDF and InferSent in clustering accuracy and coverage.
• Claims are embedded into dense vector space, and DBScan is used with a minimum cluster size of one to allow dynamic, incremental clustering.
• To improve cluster compactness and avoid spurious connections, a Louvain community detection algorithm is applied to the graph of claim similarities.
• The system supports real-time claim insertion with optimized computation, maintaining ~300ms latency per claim regardless of cluster size.
• A formula combining precision on positive claims, precision on claim clusters, and number of claims is used to evaluate clustering quality.

Experimental results

Research questions

• RQ1 Can contextual sentence embeddings effectively represent claims for real-time detection and retrieval in news articles?
• RQ2 How can claim clustering be made scalable and efficient in a dynamic, real-time setting?
• RQ3 What is the impact of different embedding models (e.g., TF-IDF, InferSent, USE Large) on clustering accuracy and coverage?
• RQ4 Can graph-based community detection improve cluster quality when using density-based clustering like DBScan?
• RQ5 To what extent can previously factchecked claims be reused for new, semantically similar claims?

Key findings

• Google’s Universal Sentence Encoder Large outperformed both TF-IDF and InferSent in clustering accuracy and claim coverage, making it the optimal embedding for this task.
• TF-IDF performed surprisingly well as a baseline, but was outperformed by neural embeddings in both precision and recall.
• The combination of DBScan with Louvain community detection significantly improved cluster compactness and reduced fragmentation.
• The system achieved a consistent ~300ms latency per claim, with minimal increase in time as the number of existing claims grew.
• The evaluation formula demonstrated that USE Large-based clustering achieved the highest score, indicating strong alignment with ground-truth article clusters.
• The method enables factcheckers to efficiently retrieve and reuse existing factchecks for new, semantically similar claims, reducing redundant work.