[论文解读] Multimodal Hate Speech Detection from Bengali Memes and Texts

Md Rezaul Karim, Sumon Kanti Dey+3 more|arXiv (Cornell University)|Apr 19, 2022

Hate Speech and Cyberbullying Detection被引用 6

一句话总结

本文提出了一种针对资源匮乏的孟加拉语的多模态仇恨言论检测框架，利用来自表情包和文本的文本与视觉数据。该研究构建了首个面向孟加拉语的多模态仇恨言论数据集，采用XLM-RoBERTa与DenseNet-161进行特征融合，取得了0.83的F1分数——优于单模态模型，其中文本模态表现最为有效，而表情包则具有中等程度的有用性。

ABSTRACT

Numerous machine learning (ML) and deep learning (DL)-based approaches have been proposed to utilize textual data from social media for anti-social behavior analysis like cyberbullying, fake news detection, and identification of hate speech mainly for highly-resourced languages such as English. However, despite having a lot of diversity and millions of native speakers, some languages like Bengali are under-resourced, which is due to a lack of computational resources for natural language processing (NLP). Similar to other languages, Bengali social media contents also include images along with texts (e.g., multimodal memes are posted by embedding short texts into images on Facebook). Therefore, only the textual data is not enough to judge them since images might give extra context to make a proper judgement. This paper is about hate speech detection from multimodal Bengali memes and texts. We prepared the only multimodal hate speech dataset for-a-kind of problem for Bengali, which we use to train state-of-the-art neural architectures (e.g., Bi-LSTM/Conv-LSTM with word embeddings, ConvNets + pre-trained language models, e.g., monolingual Bangla BERT, multilingual BERT-cased/uncased, and XLM-RoBERTa) to jointly analyze textual and visual information for hate speech detection. Conv-LSTM and XLM-RoBERTa models performed best for texts, yielding F1 scores of 0.78 and 0.82, respectively. As of memes, ResNet-152 and DenseNet-161 models yield F1 scores of 0.78 and 0.79, respectively. As for multimodal fusion, XLM-RoBERTa + DenseNet-161 performed the best, yielding an F1 score of 0.83. Our study suggests that text modality is most useful for hate speech detection, while memes are moderately useful.

研究动机与目标

• 为资源匮乏的孟加拉语仇恨言论检测解决计算资源不足与标注数据集缺乏的问题。
• 开发一种多模态框架，联合分析来自孟加拉语表情包和文本的文本与视觉内容，以提升仇恨言论检测效果。
• 评估不同深度学习架构与预训练语言模型在低资源环境下的仇恨言论检测效果。
• 提供开源计算资源，包括数据集、模型与代码，以推动孟加拉语自然语言处理研究。

提出的方法

• 构建了首个面向孟加拉语的多模态仇恨言论数据集，包含来自社交媒体表情包的标注文本与图像。
• 采用深度神经网络，包括Bi-LSTM、Conv-LSTM以及视觉模型如ResNet-152、DenseNet-161与EfficientNet-B7进行特征提取。
• 整合了预训练的单语与多语Transformer模型——Bangla BERT、mBERT-cased/uncased与XLM-RoBERTa，用于上下文文本表征。
• 通过晚期融合策略融合文本与视觉特征，结合来自自然语言处理与计算机视觉模型的嵌入表示。
• 使用标准自然语言处理指标（F1分数、精确率、召回率与Matthew相关系数，MCC）进行模型训练与评估。
• 采用事后可解释性技术分析模型决策过程，但模型可解释性仍存在局限。

实验结果

研究问题

• RQ1 相较于单模态文本方法，多模态方法在低资源语言（如孟加拉语）中的仇恨言论检测效果如何？
• RQ2 哪些预训练语言模型（如XLM-RoBERTa、mBERT、Bangla BERT）在孟加拉语仇恨言论检测中表现最佳？
• RQ3 在孟加拉语表情包中，文本模态与视觉模态对仇恨言论检测的相对贡献如何？
• RQ4 联合分析文本与图像是否能超越单模态基线模型，提升检测性能？
• RQ5 不同深度学习架构（如Conv-LSTM、DenseNet-161）在建模文本与视觉特征用于仇恨言论检测方面表现如何？

主要发现

• XLM-RoBERTa在单模态文本检测中取得了0.82的最高F1分数，优于其他Transformer模型。
• DenseNet-161在表情包的单模态视觉分析中取得0.79的F1分数，优于ResNet-152与其他CNN模型。
• XLM-RoBERTa与DenseNet-161的多模态融合在整体性能上表现最佳，F1分数达0.83，MCC为0.67。
• 文本模态被证实是仇恨言论检测中最具信息量的模态，而视觉模态则贡献适中。
• mBERT-uncased与EfficientNet-B7的组合在多语言模型融合中取得了0.82的F1分数，表现强劲。
• 尽管采用了多模态融合，但无一模型超越最佳单模态文本基线模型，表明仅依赖文本已能高度预测该任务。