[Paper Review] Neural Models for Offensive Language Detection

Ehab Hamdy|arXiv (Cornell University)|May 30, 2021

Hate Speech and Cyberbullying Detection4 references4 citations

TL;DR

This master's thesis evaluates neural network models for offensive language detection, focusing on BERT-based architectures and efficient alternatives like BlazingText. It demonstrates that RoBERTa achieves the highest F1 score on offensive language detection tasks, while BlazingText offers significantly faster inference, making it suitable for real-time applications.

ABSTRACT

Offensive language detection is an ever-growing natural language processing (NLP) application. This growth is mainly because of the widespread usage of social networks, which becomes a mainstream channel for people to communicate, work, and enjoy entertainment content. Many incidents of sharing aggressive and offensive content negatively impacted society to a great extend. We believe contributing to improving and comparing different machine learning models to fight such harmful contents is an important and challenging goal for this thesis. We targeted the problem of offensive language detection for building efficient automated models for offensive language detection. With the recent advancements of NLP models, specifically, the Transformer model, which tackled many shortcomings of the standard seq-to-seq techniques. The BERT model has shown state-of-the-art results on many NLP tasks. Although the literature still exploring the reasons for the BERT achievements in the NLP field. Other efficient variants have been developed to improve upon the standard BERT, such as RoBERTa and ALBERT. Moreover, due to the multilingual nature of text on social media that could affect the model decision on a given tween, it is becoming essential to examine multilingual models such as XLM-RoBERTa trained on 100 languages and how did it compare to unilingual models. The RoBERTa based model proved to be the most capable model and achieved the highest F1 score for the tasks. Another critical aspect of a well-rounded offensive language detection system is the speed at which a model can be trained and make inferences. In that respect, we have considered the model run-time and fine-tuned the very efficient implementation of FastText called BlazingText that achieved good results, which is much faster than BERT-based models.

Motivation & Objective

• To evaluate state-of-the-art neural models for offensive language detection in social media.
• To compare the performance of multilingual and unilingual models, particularly RoBERTa and XLM-RoBERTa.
• To assess inference speed and efficiency of models like BlazingText for real-time deployment.
• To investigate the impact of data augmentation and fine-tuning strategies on model performance.
• To provide a comprehensive benchmark of transformer-based and fast inference models on offensive language detection tasks.

Proposed method

• Employs BERT and its variants (RoBERTa, ALBERT) for sequence classification on offensive language detection.
• Utilizes transfer learning by fine-tuning pre-trained language models on offensive language datasets.
• Applies data augmentation techniques such as back-translation to improve model generalization.
• Implements BlazingText, a fast PyTorch-based model, for efficient inference and comparison with BERT-based models.
• Performs hyperparameter tuning and cross-validation to optimize model performance.
• Uses standard preprocessing pipelines including tokenization, subword tokenization, and input formatting for BERT and BlazingText.

Experimental results

Research questions

• RQ1 Which transformer-based model achieves the highest F1 score on offensive language detection tasks?
• RQ2 How does the performance of multilingual models like XLM-RoBERTa compare to unilingual models on offensive language detection?
• RQ3 What is the trade-off between model accuracy and inference speed when using BERT versus BlazingText?
• RQ4 To what extent does data augmentation improve model robustness and generalization?
• RQ5 How effective are fine-tuning strategies for BERT-based models on low-resource offensive language detection scenarios?

Key findings

• RoBERTa-based models achieved the highest F1 score across both offensive language detection tasks, outperforming BERT and other variants.
• BlazingText demonstrated significantly faster inference speed compared to BERT-based models, making it suitable for real-time applications.
• Multilingual models like XLM-RoBERTa showed strong performance, though RoBERTa outperformed it on the evaluated datasets.
• Data augmentation techniques such as back-translation improved model generalization, particularly on low-resource settings.
• Fine-tuning BERT-based models with task-specific heads yielded consistent performance gains across both binary and multi-class offensive language classification.
• The study confirms that RoBERTa is the most effective model for offensive language detection in terms of accuracy, while BlazingText is optimal for speed-critical deployments.