[Paper Review] A transformer based approach for fighting COVID-19 fake news

S. M. Sadiq-Ur-Rahman Shifath, Mohammad Faiyaz Khan+1 more|arXiv (Cornell University)|Jan 28, 2021

Misinformation and Its ImpactsSocial Sciences21 references31 citations

TL;DR

The paper builds a stacking ensemble of eight transformer-based pre-trained models with MLP heads to detect COVID-19 fake news, achieving about 0.98 accuracy on the competition test/validation set.

ABSTRACT

The rapid outbreak of COVID-19 has caused humanity to come to a stand-still and brought with it a plethora of other problems. COVID-19 is the first pandemic in history when humanity is the most technologically advanced and relies heavily on social media platforms for connectivity and other benefits. Unfortunately, fake news and misinformation regarding this virus is also available to people and causing some massive problems. So, fighting this infodemic has become a significant challenge. We present our solution for the "Constraint@AAAI2021 - COVID19 Fake News Detection in English" challenge in this work. After extensive experimentation with numerous architectures and techniques, we use eight different transformer-based pre-trained models with additional layers to construct a stacking ensemble classifier and fine-tuned them for our purpose. We achieved 0.979906542 accuracy, 0.979913119 precision, 0.979906542 recall, and 0.979907901 f1-score on the test dataset of the competition.

Motivation & Objective

• Investigate effective detection of COVID-19 fake news in English to combat the infodemic.
• Evaluate multiple transformer-based pre-trained models for text classification on the competition dataset.
• Develop an ensemble framework to leverage diverse transformer representations for improved accuracy.

Proposed method

• Fine-tune eight transformer models (BERT, GPT-2, XLNet, RoBERTa, DistilRoBERTa, ALBERT, BART, DeBERTa) with an added MLP head per model.
• Train a meta-learner (fully connected layers) on the predictions of the individual models to produce the final output.
• Experiment with RCNN, SVM, and MLP atop transformer representations, and compare against traditional models (Bi-LSTM, 1D-CNN, HAN, RCNN, AMCNN).
• Construct a stacking ensemble that uses an 1x8 feature vector from model outputs fed into the meta-learner.
• Use additional external dataset (FakeCovid) for data augmentation, then assess impact of extra data on validation performance.
• Hyper-parameter tuning across learning rates, layer sizes, and regularization to balance performance and overfitting.

Experimental results

Research questions

• RQ1 What is the performance gain when combining multiple transformer models with an MLP head for COVID-19 fake news detection?
• RQ2 How does ensemble learning compare to individual transformer models and traditional architectures on the Constraint@AAAI2021 dataset?
• RQ3 Does adding external data improve or harm validation accuracy due to class imbalance or dataset shift?
• RQ4 What is the impact of various meta-learner architectures on fusion performance of model predictions?

Key findings

• Transformer baselines (RoBERTa, BERT) significantly outperform traditional models on validation data.
• Simple top-layer classifiers on transformers yield strong results; combinations with RCNN or SVM did not surpass simple transformers.
• MLP atop RoBERTa and other models improves performance, with RoBERTa+MLP achieving high metrics.
• Ensembling eight diverse transformer models (Ensemble-v3) yields the highest validation accuracy (0.984) and F1-score (0.983).
• Ensemble-v1 achieves 0.981 accuracy; Ensemble-v2 achieves 0.983 accuracy; Ensemble-v3 achieves 0.984 accuracy on the validation set.
• Adding more data can decrease performance due to class imbalance; final model training omitted extra data to maintain generalization.