[Paper Review] Neural Network Architecture for Credibility Assessment of Textual Claims

Nurendra Choudhary, Rajat Singh+2 more|arXiv (Cornell University)|Mar 28, 2018

Misinformation and Its Impacts18 citations

TL;DR

This paper proposes CREDO, a neural network architecture that assesses the credibility of textual claims by integrating source and author credibility, semantic similarity with trusted documents via bidirectional LSTM and siamese networks, and sentiment analysis. On the Snopes dataset, CREDO outperforms state-of-the-art methods, with semantic similarity being the most influential module, contributing a 32.7% performance drop when excluded.

ABSTRACT

Text articles with false claims, especially news, have recently become aggravating for the Internet users. These articles are in wide circulation and readers face difficulty discerning fact from fiction. Previous work on credibility assessment has focused on factual analysis and linguistic features. The task's main challenge is the distinction between the features of true and false articles. In this paper, we propose a novel approach called Credibility Outcome (CREDO) which aims at scoring the credibility of an article in an open domain setting. CREDO consists of different modules for capturing various features responsible for the credibility of an article. These features includes credibility of the article's source and author, semantic similarity between the article and related credible articles retrieved from a knowledge base, and sentiments conveyed by the article. A neural network architecture learns the contribution of each of these modules to the overall credibility of an article. Experiments on Snopes dataset reveals that CREDO outperforms the state-of-the-art approaches based on linguistic features.

Motivation & Objective

• To address the growing challenge of misinformation in news and online content by developing an automated system for credibility assessment.
• To improve upon existing approaches that rely solely on linguistic features by incorporating external knowledge and multi-source credibility signals.
• To design a modular neural network architecture that fuses diverse credibility indicators into a unified credibility score.
• To evaluate the relative contribution of each component (semantic similarity, sentiment, source/author trust) in determining overall credibility.

Proposed method

• Employ keyword extraction to generate queries for retrieving relevant documents from a knowledge base (e.g., Wikipedia, news sites).
• Use a bidirectional LSTM with a siamese network architecture to learn semantic similarity between the input article and retrieved credible documents.
• Integrate author credibility scores (ACS) and website trust scores (WTS) as features to assess source reliability.
• Apply a pre-trained sentiment analysis tool to evaluate the neutrality of the article, as factual content tends to be more neutral.
• Combine all module outputs into a unified neural network classifier (SVM-RBF or MLP-NN) to predict the final credibility score.
• Train and evaluate using 5-fold cross-validation on the Snopes dataset, with performance measured via accuracy, F1, and Pearson correlation.

Experimental results

Research questions

• RQ1 How does the integration of semantic similarity with external knowledge bases improve credibility assessment compared to linguistic-only models?
• RQ2 What is the relative contribution of each component (semantic similarity, sentiment, author/source trust) to the overall credibility prediction?
• RQ3 Can a neural network architecture effectively combine heterogeneous credibility signals into a robust, generalizable model?
• RQ4 How do different classifiers (e.g., SVM-RBF, MLP-NN) perform on the credibility classification task with this feature set?

Key findings

• CREDO outperforms state-of-the-art approaches on the Snopes dataset across all evaluation metrics, demonstrating superior generalization in open-domain credibility assessment.
• Excluding the semantic similarity module caused a 32.7% drop in accuracy, establishing it as the most critical component of the system.
• The sentiment analysis module contributed a 7.1% improvement in accuracy, indicating a measurable but secondary role compared to semantic similarity.
• Author credibility scores (ACS) and website trust scores (WTS) each contributed a minor, ~1% improvement in accuracy, suggesting limited standalone impact.
• SVM with RBF kernel and Multi-Layer Perceptron (MLP-NN) classifiers achieved the best performance, indicating that non-linear decision boundaries are essential for this task.
• The semantic similarity module achieved strong performance on the SemEval-STS 2016 benchmark, confirming its effectiveness in measuring semantic relatedness between articles.