[Paper Review] Offensive Language Identification in Greek

Zeses Pitenis, Marcos Zampieri+1 more|arXiv (Cornell University)|Mar 16, 2020

Hate Speech and Cyberbullying DetectionComputer Science29 references78 citations

TL;DR

This paper presents OGTD, the first Greek annotated dataset for offensive language identification, and compares multiple classical and deep learning models, with LSTM/GRU with Attention achieving the best macro-F1 around 0.89 on OGTD v1.0.

ABSTRACT

As offensive language has become a rising issue for online communities and social media platforms, researchers have been investigating ways of coping with abusive content and developing systems to detect its different types: cyberbullying, hate speech, aggression, etc. With a few notable exceptions, most research on this topic so far has dealt with English. This is mostly due to the availability of language resources for English. To address this shortcoming, this paper presents the first Greek annotated dataset for offensive language identification: the Offensive Greek Tweet Dataset (OGTD). OGTD is a manually annotated dataset containing 4,779 posts from Twitter annotated as offensive and not offensive. Along with a detailed description of the dataset, we evaluate several computational models trained and tested on this data.

Motivation & Objective

• Introduce OGTD, the first Greek annotated dataset for offensive language detection on Twitter.
• Provide a detailed dataset description, annotation guidelines, and inter-annotator reliability.
• Benchmark a range of classical ML and deep learning models on OGTD to establish baselines and identify effective approaches.
• Explore feature sets including TF-IDF unigrams/bigrams and linguistic features (POS, dependency tags) for Greek.
• Discuss implications for cross-lingual and Greek NLP resource development and future dataset improvements.

Proposed method

• Collect Greek tweets via Twitter API using Greece-relevant keywords and expletives to build a diverse offensive language corpus.
• Pre-process data by lowercasing and non-accent normalization; remove URLs, Emojis, and @USER mentions; deduplicate tweets.
• Annotate 4,779 tweets with Offensive/Not Offensive/Spam using Greek guidelines; compute Cohen’s kappa for inter-annotator agreement.
• Train and evaluate multiple models using TF-IDF unigram/bigram features and a range of classifiers (Linear SVM, RBF SVM, SGDC, Multinomial NB, Bernoulli NB).
• Incorporate linguistic features (POS tags, dependency relations) and word embeddings (Greek) for deep learning models; compare with multilingual BERT.
• Deep learning models include Pooled GRU, Stacked LSTM with Attention, LSTM/GRU with Attention, 2D Convolution, GRU with Capsule, LSTM with Capsule and Attention, and BERT-based approaches.

Experimental results

Research questions

• RQ1 Can a Greek offensive language dataset be created and annotated with reliable inter-annotator agreement?
• RQ2 Which feature sets (unigram vs. bigram TF-IDF, POS/dependencies, semantic embeddings) yield the best performance for Greek offensive language detection?
• RQ3 How do classical ML models compare to deep learning models on the OGTD task?
• RQ4 What is the impact of using Greek word embeddings versus multilingual BERT for this task?
• RQ5 How can OGTD be extended to v2.0 and used for cross-lingual or shared tasks like OffensEval 2020?

Key findings

• OGTD v1.0 contains 4,779 tweets with about 29% offensive content.
• Linear SVM and SGDC achieve strong macro-F1 among classical models, with Linear SVM often slightly better.
• TF-IDF unigram features generally outperform bigrams for classical models; POS and dependency features offer marginal gains for Linear SVM but hurt some other models.
• Deep learning models with Greek word embeddings outperform classical models; LSTM/GRU with Attention yields the best macro-F1 (~0.89), while multilingual BERT underperforms monolingual Greek embeddings.
• BERT-based multilingual model is less effective than Greek word embeddings for this task.