[Paper Review] Automatic Detection of Fake News

Verónica Pérez‐Rosas, Bennett Kleinberg+2 more|arXiv (Cornell University)|Aug 23, 2017

Misinformation and Its Impacts13 references377 citations

TL;DR

This paper introduces two fake news datasets across multiple domains and presents linguistic-feature–based detectors achieving substantial accuracy, with a human baseline showing comparable or better performance in some domains.

ABSTRACT

The proliferation of misleading information in everyday access media outlets such as social media feeds, news blogs, and online newspapers have made it challenging to identify trustworthy news sources, thus increasing the need for computational tools able to provide insights into the reliability of online content. In this paper, we focus on the automatic identification of fake content in online news. Our contribution is twofold. First, we introduce two novel datasets for the task of fake news detection, covering seven different news domains. We describe the collection, annotation, and validation process in detail and present several exploratory analysis on the identification of linguistic differences in fake and legitimate news content. Second, we conduct a set of learning experiments to build accurate fake news detectors. In addition, we provide comparative analyses of the automatic and manual identification of fake news.

Motivation & Objective

• Motivate the need for computational tools to assess reliability of online news across domains.
• Provide two novel fake news datasets (crowdsourced and web-derived) with ground-truth annotations.
• Explore linguistic differences between fake and legitimate news using lexical, syntactic, readability, and psycholinguistic features.
• Build and evaluate fake news detectors using machine learning to establish baselines and compare with human performance.

Proposed method

• Construct two fake-news datasets: FakeNewsAMT (crowdsourced) and Celebrity (web-derived) with paired fake/real articles.
• Extract a broad feature set including N-grams (tf-idf), punctuation (LIWC-based), LIWC semantic/psycholinguistic categories, readability metrics, and syntax (CFG-based) features.
• Train linear SVM classifiers with 5-fold cross-validation using different feature subsets and evaluate with accuracy, precision, recall, and F1.
• Perform ablation to identify best-performing feature sets per dataset (e.g., Readability for FakeNewsAMT; Punctuation and LIWC variants for Celebrity).
• Conduct cross-domain and cross-domain-domain experiments to assess generalization of detectors.
• Compare automatic detector performance with human annotators on both datasets.

Experimental results

Research questions

• RQ1 Can two multi-domain fake news datasets (crowdsourced and web-derived) support effective automatic detection of fake news?
• RQ2 Which linguistic feature families (lexical, syntactic, readability, psycholinguistic) best discriminate fake from legitimate news across domains?
• RQ3 How does cross-domain transfer perform between datasets and across news domains?
• RQ4 How does machine performance compare to human ability in judging fake vs. legitimate news?

Key findings

• Detectors using linguistic features achieve accuracies well above random (0.50) across datasets.
• In FakeNewsAMT, Readability features yield the best performance, with accuracy substantially higher than baselines; combining all linguistic features also performs strongly.
• In the Celebrity web dataset, Punctuation features provide the strongest individual performance, followed by N-grams, complete LIWC, and Syntax features.
• Cross-domain experiments show notable performance drops compared to within-domain results, indicating domain-dependent deception cues.
• Across domains, politics, education, and technology show relatively robust cross-domain performance, while sports, business, and entertainment are more domain-specific.
• Compared to human judges, automatic detectors are competitive and even outperform humans in some domains (Celebrity domain).