[Paper Review] Detecting Hate Speech and Offensive Language on Twitter using Machine Learning: An N-gram and TFIDF based Approach

Aditya Gaydhani, Vikrant Doma+2 more|arXiv (Cornell University)|Sep 23, 2018

Hate Speech and Cyberbullying Detection6 references95 citations

TL;DR

The paper builds a three-class classifier (hateful, offensive, clean) for Twitter data using n-gram TFIDF features and compares Logistic Regression, Naive Bayes, and SVM, with Logistic Regression achieving the best test accuracy of 95.6%.

ABSTRACT

Toxic online content has become a major issue in today's world due to an exponential increase in the use of internet by people of different cultures and educational background. Differentiating hate speech and offensive language is a key challenge in automatic detection of toxic text content. In this paper, we propose an approach to automatically classify tweets on Twitter into three classes: hateful, offensive and clean. Using Twitter dataset, we perform experiments considering n-grams as features and passing their term frequency-inverse document frequency (TFIDF) values to multiple machine learning models. We perform comparative analysis of the models considering several values of n in n-grams and TFIDF normalization methods. After tuning the model giving the best results, we achieve 95.6% accuracy upon evaluating it on test data. We also create a module which serves as an intermediate between user and Twitter.

Motivation & Objective

• Motivate automated detection of toxic language on Twitter to differentiate hate speech from offensive language and benign content.
• Develop a pipeline that combines data from multiple public datasets and Twitter API to train and evaluate classifiers.
• Systematically compare n-gram ranges and TFIDF normalization with multiple classifiers to identify effective feature-classifier combinations.
• Tune hyperparameters to maximize cross-validation performance and report final test results.

Proposed method

• Extract n-gram features (unigram to trigram) from tweets weighted by TFIDF.
• Evaluate L1 and L2 TFIDF normalization and three classifiers: Naive Bayes, Logistic Regression, and SVM.
• Perform grid search over feature parameters and 10-fold cross-validation to compare models.
• Tune Naive Bayes with smoothing parameter alpha and Logistic Regression with regularization C and solvers.
• Select best model based on cross-validation, then report test performance and analyze errors.

Experimental results

Research questions

• RQ1 Can a three-class classifier reliably distinguish hateful, offensive, and clean tweets using TFIDF-weighted n-gram features?
• RQ2 Which classifier (NB, LR, SVM) and which feature settings yield the highest cross-validation and test performance?
• RQ3 How do TFIDF normalization and n-gram range affect detection performance for hate speech and offensive language?
• RQ4 What are the common misclassification patterns and potential improvements for recall on the offensive class and precision on the hateful class?

Key findings

• Logistic Regression with 1-3 n-grams and L2 TFIDF normalization achieves the highest cross-validation performance among the three models and outperforms others after tuning.
• Naive Bayes with alpha = 0.1 and L2 TFIDF achieves 93.4% cross-validation accuracy, surpassing its initial results and closely competing with LR.
• Final evaluated Logistic Regression model on test data yields 95.6% accuracy with n-gram range 1-3 and TFIDF normalization L2 (C = 100, liblinear).
• In the test set, hateful, offensive, and clean classes achieve precision/recall around 0.94–0.98 ranges, with offensive recall at 0.93 and the confusion matrix indicating 4.8% of offensive tweets misclassified as hateful.
• The error analysis suggests improvements by increasing offensive class recall and reducing hateful misclassifications, and incorporating linguistic features.