[Paper Review] Constraint 2021: Machine Learning Models for COVID-19 Fake News Detection Shared Task

Thomas Felber|arXiv (Cornell University)|Jan 11, 2021

Misinformation and Its ImpactsSocial Sciences20 references22 citations

TL;DR

This paper presents a classical machine learning approach for classifying COVID-19-related social media posts as fake or real using linguistic features such as n-grams, readability, emotional tone, and punctuation. The best-performing model, a linear SVM with comprehensive pre-processing and feature engineering, achieved a 95.19% weighted F1 score on the test set, ranking 80th out of 167 submissions in the shared task.

ABSTRACT

In this system paper we present our contribution to the Constraint 2021 COVID-19 Fake News Detection Shared Task, which poses the challenge of classifying COVID-19 related social media posts as either fake or real. In our system, we address this challenge by applying classical machine learning algorithms together with several linguistic features, such as n-grams, readability, emotional tone and punctuation. In terms of pre-processing, we experiment with various steps like stop word removal, stemming/lemmatization, link removal and more. We find our best performing system to be based on a linear SVM, which obtains a weighted average F1 score of 95.19% on test data, which lands a place in the middle of the leaderboard (place 80 of 167).

Motivation & Objective

• To address the challenge of classifying misinformation in social media during the COVID-19 pandemic.
• To evaluate the effectiveness of classical machine learning models in detecting fake news using diverse linguistic features.
• To optimize pre-processing pipelines and feature combinations for improved classification performance.
• To participate in and contribute to the Constraint 2021 shared task on fake news detection in English.
• To benchmark performance against a baseline F1 score of 93.46% using a manually annotated dataset.

Proposed method

• The system employs classical machine learning models: linear SVM, logistic regression, random forest, naive Bayes, and multilayer perceptron.
• Text pre-processing includes lowercase transformation, link removal, stop word removal, reply removal, XML entity replacement, and lemmatization using NLTK’s TweetTokenizer.
• Linguistic features are extracted from pre-processed text, including unigrams and bigrams (n-grams), Flesch Reading Ease (readability), punctuation counts (AllPunc, QMark, Exclam), and emotional tone (Tone, affect, social, Authentic) via LIWC.
• A grid search is conducted over hyperparameters and feature combinations to identify the optimal configuration for each model.
• The best-performing model is selected based on validation set performance and used for final test set predictions.
• Five system runs are submitted, one per model, with the linear SVM achieving the highest test F1 score.

Experimental results

Research questions

• RQ1 Which classical machine learning model performs best for classifying COVID-19 fake news in social media posts?
• RQ2 How do different combinations of linguistic features (n-grams, readability, emotional tone, punctuation) impact classification performance?
• RQ3 What is the optimal pre-processing pipeline for enhancing fake news detection accuracy?
• RQ4 To what extent do feature engineering and model selection improve performance over the baseline F1 score of 93.46%?
• RQ5 How does the inclusion of multiple linguistic feature types compare to using individual features in isolation?

Key findings

• The linear SVM model achieved the highest weighted average F1 score of 95.70% on the validation set, outperforming the baseline of 93.46%.
• The best-performing system, using a linear SVM with full feature and pre-processing pipeline, achieved a 95.19% F1 score on the test set, ranking 80th out of 167 participants.
• N-gram features with comprehensive pre-processing alone achieved a 94.89% F1 score, indicating strong performance even without additional features.
• Combining readability, punctuation, and emotional tone features individually yielded poor performance (57.58%, 49.17%, and 59.22% F1, respectively), but their combination improved to 67.21%.
• Only the linear SVM and logistic regression models surpassed the baseline F1 score of 93.46% on the validation set.
• The confusion matrix of the best SVM model showed balanced precision and recall, indicating robust generalization on the validation data.