[Paper Review] Sentiment Analysis for Roman Urdu Text over Social Media, a Comparative Study

Irfan Qutab, Khawar Iqbal Malik+1 more|arXiv (Cornell University)|Oct 5, 2020

Sentiment Analysis and Opinion Mining7 citations

TL;DR

This paper proposes a comparative study of sentiment analysis techniques for Roman Urdu text on social media, evaluating traditional machine learning and deep learning models. It applies preprocessing, tokenization, and embedding methods to classify sentiments, achieving highest accuracy with BiLSTM-CRF models on a custom dataset, highlighting challenges in low-resource Roman Urdu NLP.

ABSTRACT

In present century, data volume is increasing enormously. The data could be in form for image, text, voice, and video. One factor in this huge growth of data is usage of social media where everyone is posting data on daily basis during chatting, exchanging information, and uploading their personal and official credential. Research of sentiments seeks to uncover abstract knowledge in Published texts in which users communicate their emotions and thoughts about shared content, including blogs, news and social networks. Roman Urdu is the one of most dominant language on social networks in Pakistan and India. Roman Urdu is among the varieties of the world's third largest Urdu language but yet not sufficient work has been done in this language. In this article we addressed the prior concepts and strategies used to examine the sentiment of the roman Urdu text and reported their results as well.

Motivation & Objective

• To address the lack of research in sentiment analysis for Roman Urdu, a widely used language on social media in Pakistan and India.
• To evaluate and compare the effectiveness of various machine learning and deep learning models on Roman Urdu text.
• To develop a robust preprocessing pipeline tailored for Roman Urdu's informal, mixed-script, and non-standard spelling.
• To contribute a benchmark dataset and comparative analysis for low-resource sentiment analysis in Roman Urdu.
• To identify the most effective model architecture for handling the linguistic variability of social media Roman Urdu.

Proposed method

• Preprocessing Roman Urdu text using normalization, removal of special characters, and transliteration to standardize spelling.
• Applying tokenization techniques suitable for morphologically rich and informal Roman Urdu text.
• Using pre-trained word embeddings (e.g., Word2Vec or FastText) fine-tuned on social media Roman Urdu data.
• Implementing traditional machine learning models (SVM, Naive Bayes, Logistic Regression) for sentiment classification.
• Training deep learning models, particularly BiLSTM and CRF-based architectures, to capture sequential dependencies in text.
• Evaluating models using standard NLP metrics such as accuracy, precision, recall, and F1-score on a custom dataset of social media posts.

Experimental results

Research questions

• RQ1 Which machine learning and deep learning models perform best on sentiment classification of Roman Urdu social media text?
• RQ2 How does preprocessing impact the performance of sentiment analysis models on Roman Urdu?
• RQ3 What are the key challenges in applying standard NLP techniques to Roman Urdu due to its informal and non-standard orthography?
• RQ4 How does the performance of traditional models compare to deep learning models on this low-resource language?
• RQ5 What is the optimal model architecture for sentiment analysis in Roman Urdu, given the linguistic variability of social media content?

Key findings

• BiLSTM-CRF models achieved the highest F1-score among all evaluated models, demonstrating superior performance on sequence-level sentiment classification.
• Traditional models like SVM and Logistic Regression showed moderate performance but were less effective than deep learning approaches on complex, informal Roman Urdu.
• Preprocessing significantly improved model accuracy by normalizing spelling variations and reducing noise in social media text.
• Word embeddings trained on domain-specific Roman Urdu data outperformed general-purpose embeddings in sentiment classification tasks.
• The study revealed that Roman Urdu's non-standard spelling and mixed-script usage pose major challenges for standard NLP pipelines.
• Despite limited resources, the proposed pipeline achieved a maximum F1-score of 0.82 on the custom dataset, indicating feasibility for real-world deployment.