[Paper Review] BanFakeNews: A Dataset for Detecting Fake News in Bangla

Zobaer Hossain, Ashraful Rahman+2 more|arXiv (Cornell University)|Apr 19, 2020

Misinformation and Its Impacts47 references70 citations

TL;DR

Introduces BanFakeNews, a ~50K Bangla fake news dataset with annotations and benchmarks using lexical, syntactic, semantic features and transformer models. Provides human baselines and a comparative analysis against state-of-the-art NLP techniques.

ABSTRACT

Observing the damages that can be done by the rapid propagation of fake news in various sectors like politics and finance, automatic identification of fake news using linguistic analysis has drawn the attention of the research community. However, such methods are largely being developed for English where low resource languages remain out of the focus. But the risks spawned by fake and manipulative news are not confined by languages. In this work, we propose an annotated dataset of ~50K news that can be used for building automated fake news detection systems for a low resource language like Bangla. Additionally, we provide an analysis of the dataset and develop a benchmark system with state of the art NLP techniques to identify Bangla fake news. To create this system, we explore traditional linguistic features and neural network based methods. We expect this dataset will be a valuable resource for building technologies to prevent the spreading of fake news and contribute in research with low resource languages.

Motivation & Objective

• Address the lack of Bangla fake news resources by releasing a publicly available annotated dataset (~50K articles).
• Benchmark Bangla fake news detection using a wide range of linguistic features and neural architectures.
• Assess human performance on Bangla fake news detection and analyze factors influencing accuracy.
• Provide insights on feature effectiveness (lexical, syntactic, semantic, metadata) and pre-trained language models for Bangla.
• Encourage future expansion to 50K labeled examples and inclusion of additional features (source metadata).

Proposed method

• Assemble a Bangla fake news dataset including authentic, fake, clickbait, satire, and misleading content with headlines, articles, domain and metadata.
• Extract traditional linguistic features: word and character n-grams with TF-IDF; POS tag frequencies; word embeddings (FastText Bangla and News embeddings); punctuation and metadata (site Alexa rank, article lengths).
• Train traditional machine learning models (SVM, RF, LR) on linguistic features; perform 70:30 train-test split with 10% held-out validation.
• Experiment with neural nets (CNN, Bi-LSTM with attention) and pre-trained multilingual BERT for text classification; tune with Adam, learning rate 2e-5, batch size 32; evaluate with Micro-F1 and class-specific F1 for the fake class.
• Use a multilingual BERT model via HuggingFace Transformers for baseline comparison on the Bangla dataset.

Experimental results

Research questions

• RQ1 What are the most informative Bangla linguistic features for fake news detection?
• RQ2 Can traditional lexical/syntactic/semantic features outperform neural models on Bangla fake news?
• RQ3 How does a pre-trained multilingual transformer (BERT) perform on Bangla fake news detection compared to linear classifiers?
• RQ4 What is the human performance baseline on Bangla fake news detection for this dataset?
• RQ5 How does metadata (source, headline-text relation, site popularity) influence detection performance?

Key findings

• Peak F1 for the fake class with all linguistic features reach 0.91 using SVM-based models.
• Lexical and character n-gram features generally outperform other feature sets for fake news detection in Bangla.
• Neural models (CNN, Bi-LSTM, BERT) improve over baselines but do not surpass the best linear classifier using all linguistic features (BERT achieves ~0.68 fake-F1 while All Features with SVM reaches 0.91).
• The human baseline yielded fake-class F1 around 0.58–0.70 depending on annotator, with inter-annotator agreement (Fleiss’ Kappa) ~0.389.
• Character-level features are highlighted as particularly effective, suggesting integration into neural architectures in future work.
• The dataset includes annotated source metadata, headline-article relation, and site popularity as potential predictive features.