[Paper Review] Some Like it Hoax: Automated Fake News Detection in Social Networks

Eugenio Tacchini, Gabriele Ballarin+3 more|arXiv (Cornell University)|Apr 25, 2017

Misinformation and Its Impacts14 references118 citations

TL;DR

The paper shows that hoax classification can be done from the set of users who like posts, using logistic regression and a harmonic crowdsourcing approach, achieving over 99% accuracy on a Facebook dataset with minimal labeled data.

ABSTRACT

In recent years, the reliability of information on the Internet has emerged as a crucial issue of modern society. Social network sites (SNSs) have revolutionized the way in which information is spread by allowing users to freely share content. As a consequence, SNSs are also increasingly used as vectors for the diffusion of misinformation and hoaxes. The amount of disseminated information and the rapidity of its diffusion make it practically impossible to assess reliability in a timely manner, highlighting the need for automatic hoax detection systems. As a contribution towards this objective, we show that Facebook posts can be classified with high accuracy as hoaxes or non-hoaxes on the basis of the users who "liked" them. We present two classification techniques, one based on logistic regression, the other on a novel adaptation of boolean crowdsourcing algorithms. On a dataset consisting of 15,500 Facebook posts and 909,236 users, we obtain classification accuracies exceeding 99% even when the training set contains less than 1% of the posts. We further show that our techniques are robust: they work even when we restrict our attention to the users who like both hoax and non-hoax posts. These results suggest that mapping the diffusion pattern of information can be a useful component of automatic hoax detection systems.

Motivation & Objective

• Motivate automatic hoax detection due to rapid misinformation spread on social networks.
• Investigate whether the audience that likes a post reveals its hoax status.
• Develop two classifiers that operate on user-post interaction data.
• Evaluate scalability and transferability of the methods across pages and communities.

Proposed method

• Represent each post by the binary vector of users who liked it and apply logistic regression to learn user weights for hoax/non-hoax prediction.
• Adapt boolean label crowdsourcing (harmonic algorithm) to a setting with a labeled training set, modeling likes as positive votes and updating alpha/beta parameters to infer post legitimacy.
• Use a bipartite graph of posts and users with iterative updates to propagate information from labeled posts to unlabeled posts.
• In logistic regression, weights w_u encode each user's tendency to like non-hoax vs hoax posts.
• In harmonic BLC, initialize known-hoax and known-non-hoax posts, then iteratively update user and post beliefs via alpha/beta counts.

Experimental results

Research questions

• RQ1 Can hoaxes be identified based on the set of users who interact with them (like them)?
• RQ2 How does classification accuracy scale with the size of the manually labeled training set?
• RQ3 How well do the methods transfer information across different Facebook pages (communities)?
• RQ4 How robust are the methods when considering relatively mixed user communities (intersection dataset)?

Key findings

• Both methods achieve high accuracy on the complete dataset, exceeding 99% with surprisingly small training sets.
• The harmonic BLC method yields near-perfect cross-page transfer accuracy (≈99%+) when learning from other pages, even with limited labeled data.
• On the intersection dataset, logistic regression outperforms harmonic BLC when training data is small, with about 90% accuracy at 10% training.
• Harmonic BLC can classify posts with about 0.5% of posts labeled (≈80 posts) achieving >99% accuracy on the complete dataset.
• The approaches demonstrate robustness to polarization and overlapping user communities across hoax and non-hoax pages.