[Paper Review] Sentiment Analysis for YouTube Comments in Roman Urdu

Tooba Tehreem|arXiv (Cornell University)|Feb 19, 2021

Sentiment Analysis and Opinion Mining23 references4 citations

TL;DR

This study proposes a supervised sentiment analysis framework for Roman Urdu YouTube comments using five machine learning models. It applies bag-of-words feature extraction and evaluates classifiers on a 14,131-sample dataset, finding that SVM achieves the highest accuracy of 64%, outperforming Naïve Bayes, Linear Regression, KNN, and MLP.

ABSTRACT

Sentiment analysis is a vast area in the Machine learning domain. A lot of work is done on datasets and their analysis of the English Language. In Pakistan, a huge amount of data is in roman Urdu language, it is scattered all over the social sites including Twitter, YouTube, Facebook and similar applications. In this study the focus domain of dataset gathering is YouTube comments. The Dataset contains the comments of people over different Pakistani dramas and TV shows. The Dataset contains multi-class classification that is grouped The comments into positive, negative and neutral sentiment. In this Study comparative analysis is done for five supervised learning Algorithms including linear regression, SVM, KNN, Multi layer Perceptron and Naïve Bayes classifier. Accuracy, recall, precision and F-measure are used for measuring performance. Results show that accuracy of SVM is 64 percent, which is better than the rest of the list.

Motivation & Objective

• To enable automated sentiment classification of user comments in Roman Urdu, a widely used informal script in Pakistan.
• To address the scarcity of labeled datasets and models for sentiment analysis in Roman Urdu, a low-resource language variant.
• To compare the performance of five supervised learning algorithms—Naïve Bayes, Linear Regression, SVM, KNN, and MLP—on a multiclass (positive, negative, neutral) Roman Urdu comment dataset.
• To evaluate model performance using standard metrics: accuracy, precision, recall, and F1-score across 10-fold cross-validation.
• To provide a baseline for future research in low-resource, script-specific sentiment analysis in South Asian languages.

Proposed method

• A labeled dataset of 14,131 YouTube comments in Roman Urdu was collected from Kaggle, with sentiment labels (positive, negative, neutral) manually assigned.
• Text preprocessing included removing null values and tokenizing comments into word features for vectorization.
• The bag-of-words (BoW) technique was used to extract 3,000 most frequent features from the text corpus.
• The dataset was split into 90% training and 10% testing sets using 10-fold cross-validation to ensure robust evaluation.
• Five supervised classifiers—Naïve Bayes, Linear Regression, SVM, K-Nearest Neighbors (KNN), and Multilayer Perceptron (MLP)—were trained and evaluated.
• Performance was measured using confusion matrices and standard metrics: accuracy, precision, recall, and F1-score, with results averaged over 10 runs.

Experimental results

Research questions

• RQ1 Which supervised machine learning algorithm performs best for sentiment classification of Roman Urdu YouTube comments?
• RQ2 How does the performance of traditional classifiers (e.g., SVM, Naïve Bayes) compare to neural networks (e.g., MLP) on low-resource Roman Urdu text?
• RQ3 What is the impact of feature selection (bag-of-words with 3,000 features) on model accuracy in sentiment analysis of informal Roman Urdu?
• RQ4 How balanced is the sentiment distribution in the dataset, and does class imbalance affect model generalization?
• RQ5 Can a multiclass sentiment classification model achieve acceptable performance on a low-resource language like Roman Urdu with limited labeled data?

Key findings

• SVM achieved the highest accuracy of 64% on the test set, outperforming all other models in the comparative evaluation.
• Naïve Bayes showed moderate performance with an accuracy below 60%, indicating limitations in handling the linguistic complexity of Roman Urdu.
• Linear Regression performed poorly, with accuracy significantly lower than SVM and KNN, suggesting it is unsuitable for this multiclass NLP task.
• KNN and MLP showed comparable performance, both outperforming Linear Regression but underperforming SVM, with F1-scores around 0.55–0.58.
• The dataset was imbalanced, with 48% neutral, 29% positive, and 24% negative comments, which may have influenced model bias and generalization.
• Despite the low overall accuracy, SVM demonstrated the most stable performance across all evaluation metrics, especially in recall and F1-score for the positive and neutral classes.