[Paper Review] Overview of the Shared Task on Fake News Detection in Urdu at FIRE 2020

Maaz Amjad, Butt, Sabur+4 more|arXiv (Cornell University)|Jul 11, 2022

Misinformation and Its Impacts4 citations

TL;DR

This paper presents the second shared task on fake news detection in Urdu, organized at FIRE 2021, featuring a dataset of 1,300 annotated news articles across five domains. The best-performing system, using linear SVM with SGD optimization, achieved an F1-macro score of 0.679, outperforming deep neural networks and highlighting the effectiveness of classical machine learning approaches on this low-resource language task.

ABSTRACT

Automatic detection of fake news is a highly important task in the contemporary world. This study reports the 2nd shared task called UrduFake@FIRE2021 on identifying fake news detection in Urdu. The goal of the shared task is to motivate the community to come up with efficient methods for solving this vital problem, particularly for the Urdu language. The task is posed as a binary classification problem to label a given news article as a real or a fake news article. The organizers provide a dataset comprising news in five domains: (i) Health, (ii) Sports, (iii) Showbiz, (iv) Technology, and (v) Business, split into training and testing sets. The training set contains 1300 annotated news articles -- 750 real news, 550 fake news, while the testing set contains 300 news articles -- 200 real, 100 fake news. 34 teams from 7 different countries (China, Egypt, Israel, India, Mexico, Pakistan, and UAE) registered to participate in the UrduFake@FIRE2021 shared task. Out of those, 18 teams submitted their experimental results, and 11 of those submitted their technical reports, which is substantially higher compared to the UrduFake shared task in 2020 when only 6 teams submitted their technical reports. The technical reports submitted by the participants demonstrated different data representation techniques ranging from count-based BoW features to word vector embeddings as well as the use of numerous machine learning algorithms ranging from traditional SVM to various neural network architectures including Transformers such as BERT and RoBERTa. In this year's competition, the best performing system obtained an F1-macro score of 0.679, which is lower than the past year's best result of 0.907 F1-macro. Admittedly, while training sets from the past and the current years overlap to a large extent, the testing set provided this year is completely different.

Motivation & Objective

• To advance automatic fake news detection in Urdu, a low-resource language with limited NLP resources.
• To motivate the research community to develop effective methods for identifying fake news in Urdu using diverse NLP techniques.
• To provide a publicly available, multi-domain dataset of 1,300 annotated Urdu news articles for training and evaluation.
• To compare the performance of traditional machine learning models against modern deep learning and pre-trained transformer models on Urdu fake news detection.
• To investigate the effectiveness of feature engineering and model architecture choices in a low-resource setting.

Proposed method

• The task was formulated as a binary text classification problem to label news articles as real or fake.
• A dataset of 1,300 training and 300 testing articles was constructed from five domains: health, sports, showbiz, technology, and business.
• Real news were collected from credible sources like BBC Urdu News and CNN Urdu, while fake news were professionally written to mimic real misinformation.
• Participants employed diverse techniques, including bag-of-words (BoW), word embeddings, and pre-trained contextual models such as BERT, RoBERTa, MuRIL, and RoBERTa-urdu-small.
• Ensemble methods, CNNs, and various machine learning models (e.g., SVM, SVC) were used, with the best system using linear SVM optimized via Stochastic Gradient Descent.
• Evaluation was based on F1-macro, precision, recall, and accuracy, with results reported per class and overall.

Experimental results

Research questions

• RQ1 How do traditional machine learning models compare to deep neural networks in detecting fake news in low-resource Urdu?
• RQ2 What is the impact of using multilingual versus Urdu-specific pre-trained language models on fake news detection performance?
• RQ3 To what extent do hand-crafted features outperform end-to-end neural representations in this low-resource setting?
• RQ4 How generalizable are models trained on one set of fake news to a completely different test set, as in this year’s task?
• RQ5 What role does data quality and domain diversity play in the performance of fake news detection systems in Urdu?

Key findings

• The best-performing system achieved an F1-macro score of 0.679, significantly lower than the 0.907 F1-macro score from the previous year’s shared task.
• Despite the training data overlap with the prior year, the testing set was entirely new, indicating a shift in data distribution or difficulty.
• The top system used a linear SVM with Stochastic Gradient Descent, outperforming more complex models like BERT and RoBERTa.
• Classical feature-based models with SVMs demonstrated superior performance compared to end-to-end neural networks and contextual embeddings.
• The aggregated F1-macro score across all systems averaged 0.553, with a standard deviation of 0.093, indicating high variance in performance.
• Only 11 out of 18 submitting teams provided technical reports, a notable increase from the 2020 edition, suggesting growing research interest and engagement.