[Paper Review] It's Always April Fools' Day! On the Difficulty of Social Network Misinformation Classification via Propagation Features

Mauro Conti, Daniele Lain+3 more|IMT Institutional Repository (IMT School for Advanced Studies Lucca)|Jan 16, 2017

Misinformation and Its Impacts36 references4 citations

TL;DR

This paper investigates whether structural propagation features of social media content—such as cascade topology and user interaction patterns—can reliably classify misinformation on Facebook. Despite using robust, manipulation-resistant features, the study finds that classification performance remains poor (F1-score ≤ 0.65 during propagation, <0.70 at completion), indicating that misinformation and factual scientific content exhibit indistinguishable diffusion dynamics, challenging the feasibility of purely propagation-based detection systems.

ABSTRACT

Given the huge impact that Online Social Networks (OSN) had in the way people get informed and form their opinion, they became an attractive playground for malicious entities that want to spread misinformation, and leverage their effect. In fact, misinformation easily spreads on OSN and is a huge threat for modern society, possibly influencing also the outcome of elections, or even putting people's life at risk (e.g., spreading "anti-vaccines" misinformation). Therefore, it is of paramount importance for our society to have some sort of "validation" on information spreading through OSN. The need for a wide-scale validation would greatly benefit from automatic tools. In this paper, we show that it is difficult to carry out an automatic classification of misinformation considering only structural properties of content propagation cascades. We focus on structural properties, because they would be inherently difficult to be manipulated, with the the aim of circumventing classification systems. To support our claim, we carry out an extensive evaluation on Facebook posts belonging to conspiracy theories (as representative of misinformation), and scientific news (representative of fact-checked content). Our findings show that conspiracy content actually reverberates in a way which is hard to distinguish from the one scientific content does: for the classification mechanisms we investigated, classification F1-score never exceeds 0.65 during content propagation stages, and is still less than 0.7 even after propagation is complete.

Motivation & Objective

• To evaluate whether structural features of content propagation cascades can reliably classify misinformation on social networks.
• To assess whether these features—considered difficult for attackers to manipulate—can enable effective automatic misinformation detection.
• To compare the propagation dynamics of conspiracy theory posts versus fact-checked scientific news on Facebook.
• To determine whether early-stage or final-stage propagation features yield usable classification performance.
• To investigate whether echo chambers and user polarization obscure structural distinctions between content types.

Proposed method

• The study analyzes Facebook posts from scientific and conspiracy theory pages in Italy, using publicly available data via the Facebook Graph API.
• It extracts 28 structural features from propagation cascades, including topological properties (e.g., depth, width, branching factor) and user interaction patterns.
• Three classifiers—Linear Discriminant (LD), Random Forest (RF), and Multi-Layer Perceptron (MLP)—are trained on both unbalanced and undersampled balanced datasets.
• Performance is evaluated using 5-fold cross-validation with metrics including AUC, Cohen’s Kappa, precision, recall, accuracy, and F1-score.
• The analysis compares classification performance across two stages: early propagation and final propagation completion.
• The study uses undersampling to balance the dataset and reduce bias, ensuring fair evaluation of classifier performance.

Experimental results

Research questions

• RQ1 Can structural propagation features alone distinguish between misinformation and factual scientific content on Facebook?
• RQ2 Does the propagation dynamics of conspiracy theory content differ significantly from those of scientific content in terms of cascade topology?
• RQ3 Can early-stage propagation features enable reliable misinformation detection before content fully spreads?
• RQ4 To what extent do echo chambers and user polarization obscure structural differences in content diffusion?
• RQ5 Is the performance of propagation-based classifiers significantly better than random baseline classification?

Key findings

• During the early propagation stage, the best F1-score across all classifiers was 0.65, indicating limited ability to classify misinformation based on structural features alone.
• Even after propagation completion, the highest F1-score achieved was below 0.70, showing minimal improvement over random classification.
• The AUC and Cohen’s Kappa metrics remained below 0.75, with ROC curves close to the random baseline, indicating poor discriminative power.
• Random Forest (RF) and Multi-Layer Perceptron (MLP) outperformed Linear Discriminant (LD), but still failed to achieve usable classification performance.
• The results suggest that conspiracy and scientific content follow similar propagation patterns, likely due to echo chamber dynamics reinforcing similar structural cascades.
• The study concludes that propagation-based detection is insufficient without incorporating content-related features or user polarization metrics.