[Paper Review] Presence of informal language, such as emoticons, hashtags, and slang, impact the performance of sentiment analysis models on social media text?

Aadil Gani Ganie|arXiv (Cornell University)|Jan 28, 2023

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This study investigates the impact of informal language—emoticons, hashtags, and slang—on sentiment analysis models using a convolutional neural network (CNN) trained on sarcasm, sentiment, and emoticon datasets. Results show minimal performance degradation from informal language, with a slight accuracy improvement (95.37%) when emoticon data is included, indicating informal elements have limited negative impact on model performance.

ABSTRACT

This study aimed to investigate the influence of the presence of informal language, such as emoticons and slang, on the performance of sentiment analysis models applied to social media text. A convolutional neural network (CNN) model was developed and trained on three datasets: a sarcasm dataset, a sentiment dataset, and an emoticon dataset. The model architecture was held constant for all experiments and the model was trained on 80% of the data and tested on 20%. The results revealed that the model achieved an accuracy of 96.47% on the sarcasm dataset, with the lowest accuracy for class 1. On the sentiment dataset, the model achieved an accuracy of 95.28%. The amalgamation of sarcasm and sentiment datasets improved the accuracy of the model to 95.1%, and the addition of emoticon dataset has a slight positive impact on the accuracy of the model to 95.37%. The study suggests that the presence of informal language has a restricted impact on the performance of sentiment analysis models applied to social media text. However, the inclusion of emoticon data to the model can enhance the accuracy slightly.

Motivation & Objective

• To examine the effect of informal language elements like emoticons, hashtags, and slang on sentiment analysis model performance.
• To assess whether the inclusion of informal language data improves or hinders model accuracy in social media text classification.
• To evaluate the robustness of a CNN-based sentiment analysis model when exposed to diverse informal language patterns.
• To determine if combining multiple informal language datasets enhances overall model performance.

Proposed method

• A convolutional neural network (CNN) model was trained and evaluated on three distinct datasets: sarcasm, sentiment, and emoticon data.
• The model was trained on 80% of each dataset and tested on the remaining 20% to ensure consistent evaluation.
• Model performance was measured using accuracy across different classes and dataset combinations.
• The architecture remained constant across all experiments to isolate the effect of dataset composition.
• The study combined datasets incrementally—sarcasm, sentiment, and emoticon data—to assess cumulative impact on performance.
• Performance was evaluated on both individual and combined datasets to analyze the influence of informal language types.

Experimental results

Research questions

• RQ1 How does the presence of emoticons affect the accuracy of sentiment analysis models on social media text?
• RQ2 To what extent do slang and informal hashtags degrade the performance of sentiment analysis models?
• RQ3 Does incorporating emoticon-laden data improve the robustness of sentiment analysis models?
• RQ4 How does combining sarcasm and sentiment datasets influence model accuracy when informal language is present?
• RQ5 What is the relative impact of different informal language forms (emoticons, hashtags, slang) on model performance?

Key findings

• The CNN model achieved 96.47% accuracy on the sarcasm dataset, with the lowest performance observed for class 1.
• On the sentiment dataset, the model reached 95.28% accuracy, indicating strong performance on standard sentiment classification.
• Combining the sarcasm and sentiment datasets slightly reduced accuracy to 95.10%, suggesting potential data distribution challenges.
• Incorporating the emoticon dataset into the combined model improved accuracy to 95.37%, demonstrating a small but positive impact.
• The study concludes that informal language elements such as emoticons, hashtags, and slang have a restricted negative impact on sentiment analysis model performance.
• The inclusion of emoticon data provides a measurable, albeit minor, boost in model accuracy, indicating its value in training data.