[Paper Review] Explainable Tsetlin Machine framework for fake news detection with credibility score assessment

Bimal Bhattarai, Ole‐Christoffer Granmo+1 more|arXiv (Cornell University)|May 19, 2021

Misinformation and Its Impacts35 references21 citations

TL;DR

This paper proposes an explainable Tsetlin Machine (TM) framework for fake news detection that uses logical clauses to capture lexical and semantic patterns in text, enabling both global and local interpretability. The model achieves at least 5% higher accuracy than prior baselines and provides credibility scores, outperforming BERT and XLNet in F1-score while maintaining transparency through clause-level feature analysis.

ABSTRACT

The proliferation of fake news, i.e., news intentionally spread for misinformation, poses a threat to individuals and society. Despite various fact-checking websites such as PolitiFact, robust detection techniques are required to deal with the increase in fake news. Several deep learning models show promising results for fake news classification, however, their black-box nature makes it difficult to explain their classification decisions and quality-assure the models. We here address this problem by proposing a novel interpretable fake news detection framework based on the recently introduced Tsetlin Machine (TM). In brief, we utilize the conjunctive clauses of the TM to capture lexical and semantic properties of both true and fake news text. Further, we use the clause ensembles to calculate the credibility of fake news. For evaluation, we conduct experiments on two publicly available datasets, PolitiFact and GossipCop, and demonstrate that the TM framework significantly outperforms previously published baselines by at least $5\%$ in terms of accuracy, with the added benefit of an interpretable logic-based representation. Further, our approach provides higher F1-score than BERT and XLNet, however, we obtain slightly lower accuracy. We finally present a case study on our model's explainability, demonstrating how it decomposes into meaningful words and their negations.

Motivation & Objective

• To address the lack of interpretability in deep learning-based fake news detection models.
• To develop a transparent, logic-based framework that explains classification decisions using human-readable clauses.
• To enable credibility scoring for individual fake news items based on classification confidence.
• To improve detection performance while maintaining interpretability, overcoming limitations of black-box models like BERT and XLNet.
• To demonstrate how TM clauses capture both plain and negated word patterns for robust discrimination between true and fake news.

Proposed method

• The Tsetlin Machine (TM) uses conjunctive clauses to model lexical and semantic patterns in news text, learning from labeled data.
• Each clause in the TM framework consists of plain and negated literals, enabling non-monotonic reasoning for improved discrimination.
• The model classifies news by aggregating votes from active clauses in both positive and negative polarity groups.
• Credibility scores are derived from the confidence of classification votes, allowing ranking of fake news by trustworthiness.
• The framework leverages word patterns captured in clauses to provide global interpretability, showing which features contribute to classification.
• A case study analyzes top literals (plain and negated) in clauses to demonstrate local interpretability and feature relevance.

Experimental results

Research questions

• RQ1 How can a fake news detection model achieve high accuracy while remaining interpretable through logical clause representation?
• RQ2 Can a Tsetlin Machine framework effectively learn and represent both linguistic and semantic patterns in fake news using plain and negated word literals?
• RQ3 How does the credibility score based on classification confidence improve the assessment of individual fake news items?
• RQ4 To what extent does the TM framework outperform black-box models like BERT and XLNet in terms of F1-score and interpretability?
• RQ5 How do non-monotonic clauses—combining plain and negated features—enhance the model’s discrimination power and human-understandable reasoning?

Key findings

• The TM framework achieves at least 5% higher accuracy than previously published baselines on both PolitiFact and GossipCop datasets.
• The model outperforms BERT and XLNet in F1-score, demonstrating superior performance on imbalanced or nuanced detection tasks.
• The credibility score mechanism enables ranking of fake news based on classification confidence, supporting risk assessment.
• The case study reveals that clauses frequently capture negated literals (e.g., ¬trump), which enhance discrimination by contrasting fake and true news features.
• Top captured literals in clauses include high-contrast terms like 'trump', 'rumors', 'gossip', and 'source', indicating strong discriminative power.
• The model’s interpretability is demonstrated by showing that classification decisions are based on joint contributions of multiple literals within clauses, not isolated words.