[Paper Review] Sarcasm Detection in Twitter -- Performance Impact while using Data Augmentation: Word Embeddings

Alif Tri Handoyo, Hidayaturrahman Hidayaturrahman+1 more|arXiv (Cornell University)|Aug 23, 2021

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper proposes a contextual sarcasm detection model using RoBERTa with data augmentation via GloVe word embeddings to improve performance on imbalanced Twitter datasets. By increasing sarcastic examples by 20%, the method achieves a 3.2% F-score improvement (40.4% vs. 37.2%) on the iSarcasm dataset, demonstrating enhanced detection accuracy through better data balance and context learning.

ABSTRACT

Sarcasm is the use of words usually used to either mock or annoy someone, or for humorous purposes. Sarcasm is largely used in social networks and microblogging websites, where people mock or censure in a way that makes it difficult even for humans to tell if what is said is what is meant. Failure to identify sarcastic utterances in Natural Language Processing applications such as sentiment analysis and opinion mining will confuse classification algorithms and generate false results. Several studies on sarcasm detection have utilized different learning algorithms. However, most of these learning models have always focused on the contents of expression only, leaving the contextual information in isolation. As a result, they failed to capture the contextual information in the sarcastic expression. Moreover, some datasets used in several studies have an unbalanced dataset which impacting the model result. In this paper, we propose a contextual model for sarcasm identification in twitter using RoBERTa, and augmenting the dataset by applying Global Vector representation (GloVe) for the construction of word embedding and context learning to generate more data and balancing the dataset. The effectiveness of this technique is tested with various datasets and data augmentation settings. In particular, we achieve performance gain by 3.2% in the iSarcasm dataset when using data augmentation to increase 20% of data labeled as sarcastic, resulting F-score of 40.4% compared to 37.2% without data augmentation.

Motivation & Objective

• To address the challenge of detecting sarcasm in Twitter, where sarcastic expressions are often ambiguous and difficult even for humans to interpret.
• To overcome the limitations of existing models that focus only on textual content and neglect contextual information.
• To mitigate performance degradation caused by imbalanced datasets in sarcasm detection tasks.
• To explore the impact of data augmentation using GloVe embeddings on improving model generalization and F-score in sarcasm detection.
• To evaluate the effectiveness of combining contextual modeling (RoBERTa) with augmented, balanced training data for sarcasm detection.

Proposed method

• The authors employ RoBERTa, a contextual language model, to capture deep contextual representations of tweets for sarcasm detection.
• GloVe word embeddings are used to generate synthetic training examples through data augmentation, increasing the number of sarcastic samples by 20%.
• The augmentation process balances the dataset by generating additional sarcastic examples using semantic similarity and word-level transformations based on GloVe vectors.
• The model is fine-tuned on multiple datasets, including iSarcasm, to evaluate performance under varying data augmentation settings.
• Performance is evaluated using F-score, with comparisons between models trained with and without data augmentation.
• The approach combines contextual modeling with embedding-based data augmentation to improve model robustness on imbalanced sarcasm detection tasks.

Experimental results

Research questions

• RQ1 Does data augmentation using GloVe embeddings improve sarcasm detection performance on imbalanced Twitter datasets?
• RQ2 To what extent does incorporating contextual information via RoBERTa enhance sarcasm detection beyond surface-level text analysis?
• RQ3 How does increasing the proportion of sarcastic examples through augmentation affect F-score and model generalization?
• RQ4 Can embedding-based data augmentation effectively balance datasets and reduce bias in sarcasm detection models?
• RQ5 What is the comparative performance gain of the proposed method across different sarcasm detection datasets?

Key findings

• The proposed method achieved a 3.2% absolute improvement in F-score on the iSarcasm dataset, reaching 40.4% with data augmentation compared to 37.2% without it.
• Data augmentation using GloVe embeddings effectively balanced the dataset, reducing the negative impact of class imbalance on model performance.
• The integration of RoBERTa with GloVe-based augmentation significantly improved the model's ability to detect sarcasm by capturing contextual nuances.
• The performance gain was most notable in the iSarcasm dataset, where the F-score increased from 37.2% to 40.4% after 20% data augmentation on sarcastic instances.
• The results demonstrate that combining contextual modeling with embedding-based data augmentation is effective for sarcasm detection in low-resource, imbalanced settings.
• The study confirms that data augmentation using pre-trained word embeddings like GloVe can be a viable strategy to improve model generalization in sarcasm detection tasks.