[Paper Review] "Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection

William Yang Wang|arXiv (Cornell University)|May 1, 2017

Misinformation and Its Impacts199 citations

TL;DR

The paper introduces the liar dataset with 12.8K PolitiFact statements, and proposes a hybrid CNN that fuses text with speaker meta-data, achieving state-of-the-art results on fine-grained fake news detection.

ABSTRACT

Automatic fake news detection is a challenging problem in deception detection, and it has tremendous real-world political and social impacts. However, statistical approaches to combating fake news has been dramatically limited by the lack of labeled benchmark datasets. In this paper, we present liar: a new, publicly available dataset for fake news detection. We collected a decade-long, 12.8K manually labeled short statements in various contexts from PolitiFact.com, which provides detailed analysis report and links to source documents for each case. This dataset can be used for fact-checking research as well. Notably, this new dataset is an order of magnitude larger than previously largest public fake news datasets of similar type. Empirically, we investigate automatic fake news detection based on surface-level linguistic patterns. We have designed a novel, hybrid convolutional neural network to integrate meta-data with text. We show that this hybrid approach can improve a text-only deep learning model.

Motivation & Objective

• Provide a large, real-world labeled dataset for fake news detection and fact-checking research.
• Evaluate machine learning models on short, context-rich political statements.
• Explore integrating speaker/meta-data with text representations to boost detection performance.
• Benchmark and analyze the limits of surface-level linguistic features for multi-class fake-news classification.

Proposed method

• Curate a 12.8K dataset of PolitiFact statements labeled for truthfulness with rich meta-data.
• Baseline and neural models include Logistic Regression, SVM, Bi-LSTM, and CNN on text.
• Propose a hybrid Convolutional Neural Network that integrates text embeddings with meta-data via a metadata embedding, convolution, max-pooling, Bi-LSTM, and a final softmax classifier.
• Use pre-trained word2vec (Google News, 300d) for text initialization.
• Tune hyperparameters on a validation set; report accuracy as the evaluation metric.
• Compare text-only models against the text+meta-data hybrid approach to assess gains.

Experimental results

Research questions

• RQ1 How well can short statements be classified into six fine-grained truthfulness categories using surface-level linguistic features?
• RQ2 Can a neural architecture that combines text with speaker/meta-data outperform text-only models in fake news detection?
• RQ3 What is the impact of incorporating different meta-data facets (subject, speaker, job, state, party, context, history) on detection accuracy?

Key findings

• The liar dataset contains 12.8K statements with six fine-grained truth labels: pants-fire, false, barely-true, half-true, mostly-true, true.
• The dataset is designed for broad real-world contexts and provides detailed justification and source links for each label.
• Text-only CNNs outperform other baselines (accuracy 0.270 on the test set) and significantly beat SVMs (p<0.0001).
• A hybrid text+meta-data CNN improves upon text-only baselines, with the best single result achieving 0.274 test accuracy on the liar dataset when using all meta-data features.
• Several meta-data single-feature models (e.g., Text+Speaker, Text+Job) show improvements over text-only models, illustrating the benefit of metadata integration.
• Overall, combining text with comprehensive meta-data yields the strongest performance on the test set.