[Paper Review] Utilization of Multinomial Naive Bayes Algorithm and Term Frequency Inverse Document Frequency (TF-IDF Vectorizer) in Checking the Credibility of News Tweet in the Philippines

Neil Christian R. Riego, Danny Bell Villarba|arXiv (Cornell University)|May 30, 2023

Misinformation and Its Impacts4 citations

TL;DR

This study proposes a news credibility detection system for Filipino news tweets using Multinomial Naive Bayes with TF-IDF vectorization. Trained on ground-truth annotated data, the model achieves 88.98% accuracy on unseen data, though F1-score of 89.68% indicates room for improvement in reducing false positives (misclassifying fake news as real).

ABSTRACT

The digitalization of news media become a good indicator of progress and signal to more threats. Media disinformation or fake news is one of these threats, and it is necessary to take any action in fighting disinformation. This paper utilizes ground truth-based annotations and TF-IDF as feature extraction for the news articles which is then used as a training data set for Multinomial Naive Bayes. The model has an accuracy of 99.46% in training and 88.98% in predicting unseen data. Tagging fake news as real news is a concerning point on the prediction that is indicated in the F1 score of 89.68%. This could lead to a negative impact. To prevent this to happen it is suggested to further improve the corpus collection, and use an ensemble machine learning to reinforce the prediction

Motivation & Objective

• To develop a reliable automated system for detecting fake news in news tweets from the Philippines.
• To address the growing threat of disinformation in digital news media, particularly on social platforms like Twitter.
• To improve media literacy and public trust by enabling rapid credibility assessment of viral news content.
• To evaluate the effectiveness of TF-IDF feature extraction combined with Multinomial Naive Bayes for low-resource, multilingual news credibility detection.

Proposed method

• Ground truth-based annotation of news tweets to create a labeled training dataset.
• Application of Term Frequency-Inverse Document Frequency (TF-IDF) vectorization to convert textual content into numerical features.
• Training a Multinomial Naive Bayes classifier on the TF-IDF encoded features using the annotated dataset.
• Evaluating model performance on unseen test data using standard NLP metrics including accuracy, precision, recall, and F1-score.
• Analyzing prediction errors, particularly false positives where fake news is misclassified as real.
• Suggesting corpus enhancement and ensemble learning as future improvements to reduce misclassification.

Experimental results

Research questions

• RQ1 How effective is the combination of TF-IDF and Multinomial Naive Bayes in classifying the credibility of news tweets in the Philippine context?
• RQ2 What is the model's accuracy and F1-score when predicting unseen news tweets?
• RQ3 Why does the model exhibit a relatively high rate of false positives (misclassifying fake news as real)?
• RQ4 To what extent can ground-truth annotated data improve the performance of a credibility detection model in low-resource settings?
• RQ5 What methodological improvements can reduce false positive predictions in news credibility classification?

Key findings

• The model achieved a training accuracy of 99.46%, indicating strong learning on the training data.
• On unseen test data, the model attained an accuracy of 88.98%, demonstrating solid generalization performance.
• The F1-score was 89.68%, suggesting a moderate balance between precision and recall, but indicating a notable risk of false positives.
• The high false positive rate is a critical limitation, as misclassifying fake news as real could have serious societal consequences.
• The study identifies corpus quality and model ensemble techniques as key areas for future improvement to reduce false positives.
• The results validate the feasibility of using TF-IDF and Multinomial Naive Bayes for credibility detection in low-resource, multilingual settings like the Philippines.