[Paper Review] AraBERT and Farasa Segmentation Based Approach For Sarcasm and Sentiment Detection in Arabic Tweets

Anshul Wadhawan|arXiv (Cornell University)|Mar 2, 2021

Sentiment Analysis and Opinion Mining16 citations

TL;DR

This paper presents a two-stage approach for sarcasm and sentiment detection in Arabic tweets using AraBERT and AraELECTRA models fine-tuned on the ArSarcasm-v2 dataset. Preprocessing with Farasa segmentation and noise removal enhanced model performance, achieving 85.55% F1 on sarcasm detection and 76.41% F1 on sentiment analysis, ranking 7th and 4th respectively in the shared task.

ABSTRACT

This paper presents our strategy to tackle the EACL WANLP-2021 Shared Task 2: Sarcasm and Sentiment Detection. One of the subtasks aims at developing a system that identifies whether a given Arabic tweet is sarcastic in nature or not, while the other aims to identify the sentiment of the Arabic tweet. We approach the task in two steps. The first step involves pre processing the provided ArSarcasm-v2 dataset by performing insertions, deletions and segmentation operations on various parts of the text. The second step involves experimenting with multiple variants of two transformer based models, AraELECTRA and AraBERT. Our final approach was ranked seventh and fourth in the Sarcasm and Sentiment Detection subtasks respectively.

Motivation & Objective

• To address the challenge of sarcasm and sentiment detection in low-resource Arabic social media text.
• To improve model performance on Arabic sarcasm and sentiment classification by applying targeted text preprocessing.
• To evaluate the effectiveness of AraBERT and AraELECTRA models on the ArSarcasm-v2 dataset for both sarcasm and sentiment detection tasks.
• To achieve competitive performance in the EACL WANLP-2021 shared task through model fine-tuning and preprocessing optimization.

Proposed method

• Preprocessing the ArSarcasm-v2 dataset by removing HTML markup, URLs, mentions, emails, emoticons, and extra spaces.
• Applying Farasa segmentation to improve morphological representation for specific models.
• Inserting spaces around punctuation, numbers, and non-Arabic characters to enhance tokenization.
• Fine-tuning multiple variants of AraBERT and AraELECTRA models using Hugging Face's Transformers API.
• Using a learning rate of 1e-5, batch size of 40 (base) or 4 (large), and maximum sequence length of 256 across 10 epochs.
• Selecting the best-performing model variants based on validation set F1 scores for final test submission.

Experimental results

Research questions

• RQ1 How does Farasa segmentation and noise removal preprocessing affect sarcasm and sentiment detection performance on Arabic tweets?
• RQ2 Which AraBERT and AraELECTRA model variants perform best on the ArSarcasm-v2 dataset for sarcasm and sentiment classification?
• RQ3 Can transformer-based models fine-tuned on Arabic text achieve strong performance on low-resource sarcasm and sentiment detection tasks?
• RQ4 What is the impact of model size (base vs. large) and version (v0.1 to v2) on classification performance?

Key findings

• AraBERTv2-base achieved the highest F1 score of 85.55% on the sarcasm detection task (Subtask 1), ranking 7th on the private leaderboard.
• AraBERTv0.2-large achieved the highest F1 score of 76.41% on the sentiment detection task (Subtask 2), ranking 4th on the private leaderboard.
• The AraELECTRA-base-generator model performed worst across all tasks, likely due to its GAN-oriented architecture not being optimal for text classification.
• Model performance improved with larger architecture, as seen in the higher F1 scores for AraBERTv0.2-large compared to its base counterpart.
• Preprocessing steps such as URL and mention replacement and whitespace normalization significantly reduced noise and improved model generalization.
• Fine-tuning with a learning rate of 1e-5 and 10 epochs yielded optimal results across all model variants tested.