[Paper Review] A Stylometric Inquiry into Hyperpartisan and Fake News

Martin Potthast, Johannes Kiesel+3 more|arXiv (Cornell University)|Feb 18, 2017

Authorship Attribution and Profiling19 references61 citations

TL;DR

The paper analyzes writing style to distinguish hyperpartisan from mainstream news and satire, and to assess fake news detection via style using Unmasking on a BuzzFeed‑Webis corpus; findings show hyperpartisan styles are distinguishable from mainstream, left/right wings share stylistic similarities, and style alone struggles with fake news detection.

ABSTRACT

This paper reports on a writing style analysis of hyperpartisan (i.e., extremely one-sided) news in connection to fake news. It presents a large corpus of 1,627 articles that were manually fact-checked by professional journalists from BuzzFeed. The articles originated from 9 well-known political publishers, 3 each from the mainstream, the hyperpartisan left-wing, and the hyperpartisan right-wing. In sum, the corpus contains 299 fake news, 97% of which originated from hyperpartisan publishers. We propose and demonstrate a new way of assessing style similarity between text categories via Unmasking---a meta-learning approach originally devised for authorship verification---, revealing that the style of left-wing and right-wing news have a lot more in common than any of the two have with the mainstream. Furthermore, we show that hyperpartisan news can be discriminated well by its style from the mainstream (F1=0.78), as can be satire from both (F1=0.81). Unsurprisingly, style-based fake news detection does not live up to scratch (F1=0.46). Nevertheless, the former results are important to implement pre-screening for fake news detectors.

Motivation & Objective

• Investigate whether hyperpartisan news can be distinguished from mainstream news by writing style.
• Explore whether the writing style of left-wing and right-wing news is stylistically similar.
• Assess if fake news can be detected using style features alone and how satire relates to fake/real news.

Proposed method

• Apply Unmasking, a meta-learning style analysis originally for authorship verification, to compare sets of articles by orientation (left, right, mainstream).
• Extract and evaluate a broad set of style features including character n-grams, stop words, POS n-grams, readability scores, dictionary-based features, and domain-specific features like quotes and external links.
• Use feature selection to discard infrequent features and ensure cross-category comparability.
• Train random-forest classifiers on style and topic features for hyperpartisan vs mainstream, orientation prediction, and satire detection.
• Operationalize fake news by grouping mostly false and mixture-of-true-and-false articles.
• Visualize style similarity through Unmasking slope analyses to interpret cross-category stylistic closeness.

Experimental results

Research questions

• RQ1 Is there a common stylistic pattern across hyperpartisan left-wing and right-wing news?
• RQ2 Can writing style alone discriminate hyperpartisan news from mainstream news, and satire from real news?
• RQ3 Is fake news detectable by style alone, and how does satire fit into style-based detection?

Key findings

• Hyperpartisan left-wing and right-wing articles show significant stylistic similarity compared with mainstream, as evidenced by Unmasking curves.
• Style-based classifiers can distinguish hyperpartisan from mainstream news with notable accuracy and recall (best style-based hyperpartisan vs mainstream: accuracy 0.75, recall 0.89 for hyperpartisan).
• Topic-based (bag-of-words) models can outperform style models in some three-class orientation predictions, indicating topic signals matter for fine-grained classification.
• Style features enable satire detection with strong performance (accuracy 0.82, F1 0.81), and satire is stylistically distinct from both fake and real news.
• Fake news detection via style alone performs modestly (accuracy 0.55, F1 around 0.41–0.63 depending on setup), indicating style pre-screening can aid, but is not sufficient by itself.
• Satire is stylistically more distant from fake/real news, enabling reliable discrimination from the style perspective.