[Paper Review] Leveraging Multi-Source Weak Social Supervision for Early Detection of Fake News

Kai Shu, Guo‐qing Zheng+5 more|arXiv (Cornell University)|Apr 3, 2020

Misinformation and Its Impacts43 citations

TL;DR

The paper proposes MWSS, a meta-learning framework that jointly leverages limited clean labels and multiple weak social supervision signals to detect fake news early, without using social features at prediction time.

ABSTRACT

Social media has greatly enabled people to participate in online activities at an unprecedented rate. However, this unrestricted access also exacerbates the spread of misinformation and fake news online which might cause confusion and chaos unless being detected early for its mitigation. Given the rapidly evolving nature of news events and the limited amount of annotated data, state-of-the-art systems on fake news detection face challenges due to the lack of large numbers of annotated training instances that are hard to come by for early detection. In this work, we exploit multiple weak signals from different sources given by user and content engagements (referred to as weak social supervision), and their complementary utilities to detect fake news. We jointly leverage the limited amount of clean data along with weak signals from social engagements to train deep neural networks in a meta-learning framework to estimate the quality of different weak instances. Experiments on realworld datasets demonstrate that the proposed framework outperforms state-of-the-art baselines for early detection of fake news without using any user engagements at prediction time.

Motivation & Objective

• Address the challenge of early fake news detection with limited annotated data.
• Leverage multiple weak social supervision signals from user engagements.
• Develop a meta-learning framework that estimates the quality/weight of weak labels.
• Train a fake news classifier that benefits from both clean and weak supervision.
• Enable strong performance without relying on social context at prediction time.

Proposed method

• Introduce MWSS, a framework with a shared content encoder and source-specific prediction heads for clean and weak signals.
• Implement a Label Weighting Network (LWN) to assign weights to weak labels based on instance representations and weak label embeddings.
• Formulate a bi-level optimization where the outer loop optimizes LWN to improve validation performance on clean data.
• Use a joint loss combining clean-data loss and weighted weak-label losses from multiple weak sources.
• Adopt a one-step SGD approximation for meta-parameter updates and train with Adam.
• Construct weak labels from social engagements via sentiment, bias, and credibility signals and evaluate their quality.

Experimental results

Research questions

• RQ1 Can MWSS improve fake news classification by leveraging multiple weak social supervision sources?
• RQ2 How do different weak supervision sources contribute to performance, and how robust is MWSS to multiple weak signals?

Key findings

• MWSS outperforms strong baselines when combining clean data with multiple weak sources across GossipCop and PolitiFact.
• Single weak sources are less effective than multiple sources; MWSS benefits from source-specific mappings and the LWN.
• MWSS consistently achieves the best statistics among evaluated methods, including L2R and Snorkel baselines.
• RoBERTa-based encoders generally outperform CNN encoders, with MWSS still leading when using clean+weak data.
• The weight learning mechanism assigns higher importance to cleaner signals and demonstrates complementary information across weak sources.
• A higher clean-to-weak data ratio improves performance for MWSS and other baselines.