[Paper Review] Vietnamese Hate and Offensive Detection using PhoBERT-CNN and Social Media Streaming Data

Khanh Tran, An Nguyen+4 more|arXiv (Cornell University)|Jun 1, 2022

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper proposes a PhoBERT-CNN model combined with enhanced data pre-processing and EDA techniques for Vietnamese hate and offensive comment detection, achieving an F1-score of 67.46% on ViHSD and 98.45% on HSD-VLSP, while demonstrating real-time streaming capability via Apache Spark Structured Streaming for practical deployment on social media platforms.

ABSTRACT

Society needs to develop a system to detect hate and offense to build a healthy and safe environment. However, current research in this field still faces four major shortcomings, including deficient pre-processing techniques, indifference to data imbalance issues, modest performance models, and lacking practical applications. This paper focused on developing an intelligent system capable of addressing these shortcomings. Firstly, we proposed an efficient pre-processing technique to clean comments collected from Vietnamese social media. Secondly, a novel hate speech detection (HSD) model, which is the combination of a pre-trained PhoBERT model and a Text-CNN model, was proposed for solving tasks in Vietnamese. Thirdly, EDA techniques are applied to deal with imbalanced data to improve the performance of classification models. Besides, various experiments were conducted as baselines to compare and investigate the proposed model's performance against state-of-the-art methods. The experiment results show that the proposed PhoBERT-CNN model outperforms SOTA methods and achieves an F1-score of 67,46% and 98,45% on two benchmark datasets, ViHSD and HSD-VLSP, respectively. Finally, we also built a streaming HSD application to demonstrate the practicality of our proposed system.

Motivation & Objective

• To address the lack of effective pre-processing techniques for Vietnamese social media text in hate speech detection.
• To overcome data imbalance issues in Vietnamese hate speech datasets through effective data augmentation using EDA.
• To develop a high-performing, end-to-end HSD model tailored for Vietnamese, outperforming existing baselines.
• To implement a real-time streaming hate speech detection system using Apache Spark for practical deployment on social media platforms.
• To contribute a robust, scalable solution for building safer online environments in Vietnamese-language digital spaces.

Proposed method

• Proposed a two-phase pre-processing pipeline to clean Vietnamese social media comments, improving data quality and model input.
• Combined PhoBERT, a pre-trained multilingual transformer for Vietnamese, with a Text-CNN for hierarchical feature extraction and classification.
• Applied EDA (EDA) techniques to augment minority classes in imbalanced datasets, improving model generalization.
• Trained and evaluated the PhoBERT-CNN model on two benchmark datasets: ViHSD and HSD-VLSP, using F1-score and accuracy as metrics.
• Built a real-time hate speech detection system using Apache Spark Structured Streaming 3.1.1, enabling low-latency processing with average response time of 1.56 seconds per comment.
• Used YouTube Data API to stream real-time comments, which were annotated by three experienced Vietnamese NLP annotators with a Cohen’s Kappa of 0.64.

Experimental results

Research questions

• RQ1 Can a fine-tuned PhoBERT-CNN model achieve superior performance on Vietnamese hate speech detection compared to existing SOTA methods?
• RQ2 To what extent do enhanced pre-processing and EDA techniques improve model performance on imbalanced Vietnamese hate speech datasets?
• RQ3 Can a real-time streaming system based on Apache Spark effectively classify hate and offensive comments in live social media data?
• RQ4 How does the proposed system perform on real-world, streaming comment data from YouTube, and what is its latency and accuracy?
• RQ5 What is the impact of context-aware pre-processing on the robustness and generalization of hate speech detection models in low-resource Vietnamese NLP settings?

Key findings

• The proposed PhoBERT-CNN model achieved an F1-score of 67.46% on the ViHSD dataset, outperforming baseline models and prior SOTA methods.
• On the HSD-VLSP dataset, the model reached a high F1-score of 98.45%, indicating strong performance on a more balanced and high-quality benchmark.
• The pre-processing pipeline improved the PhoBERT-CNN model’s macro F1-score by 4.80% on ViHSD and 9.70% on HSD-VLSP, demonstrating its effectiveness.
• Baseline models also benefited from the pre-processing, with average F1-score improvements of 3.58% and 10.16% on ViHSD and HSD-VLSP, respectively.
• The real-time streaming system achieved 58.19% macro F1-score and 82.02% accuracy on 500 real-time YouTube comments, with an average response time of 1.56 seconds.
• The system demonstrated low-latency processing using Apache Spark Structured Streaming, achieving sub-second response times with 1ms continuity processing.