[论文解读] On the Use of Emojis to Train Emotion Classifiers

Wegdan A. Hussien, Mahmoud Al‐Ayyoub+2 more|arXiv (Cornell University)|Feb 24, 2019

Sentiment Analysis and Opinion Mining参考文献 35被引用 10

一句话总结

本文提出一种基于表情符号的远程监督方法，利用表情符号自动标注情感推文以训练情感分类器，显著降低标注成本。尽管标签质量较低，但在阿拉伯语推文中，使用更大规模的自动标注数据（规模大一个数量级）训练的SVM、朴素贝叶斯和随机森林分类器，其性能优于较小规模的手动标注数据集。

ABSTRACT

Nowadays, the automatic detection of emotions is employed by many applications in different fields like security informatics, e-learning, humor detection, targeted advertising, etc. Many of these applications focus on social media and treat this problem as a classification problem, which requires preparing training data. The typical method for annotating the training data by human experts is considered time consuming, labor intensive and sometimes prone to error. Moreover, such an approach is not easily extensible to new domains/languages since such extensions require annotating new training data. In this study, we propose a distant supervised learning approach where the training sentences are automatically annotated based on the emojis they have. Such training data would be very cheap to produce compared with the manually created training data, thus, much larger training data can be easily obtained. On the other hand, this training data would naturally have lower quality as it may contain some errors in the annotation. Nonetheless, we experimentally show that training classifiers on cheap, large and possibly erroneous data annotated using this approach leads to more accurate results compared with training the same classifiers on the more expensive, much smaller and error-free manually annotated training data. Our experiments are conducted on an in-house dataset of emotional Arabic tweets and the classifiers we consider are: Support Vector Machine (SVM), Multinomial Naive Bayes (MNB) and Random Forest (RF). In addition to experimenting with single classifiers, we also consider using an ensemble of classifiers. The results show that using an automatically annotated training data (that is only one order of magnitude larger than the manually annotated one) gives better results in almost all settings considered.

研究动机与目标

• 解决在自然语言处理中情感分类任务的训练数据手动标注成本高且可扩展性差的问题。
• 探究通过自动生成的表情符号训练数据是否能获得优于较小规模手动筛选数据集的性能。
• 评估利用表情符号作为弱标签的远程监督在阿拉伯语社交媒体文本情感分类中的有效性。
• 比较在自动标注数据上训练的多种分类器（SVM、MNB、RF）及集成方法的性能。
• 评估情感分类任务中数据量与标签质量之间的权衡。

提出的方法

• 使用阿拉伯语推文中的表情符号作为弱标签，自动为句子分配情感类别，从而实现大规模数据收集。
• 该方法将每个表情符号视为对应推文中所表达情感的代理，形成远程监督信号。
• 通过将推文与表情符号所暗示的情感标签相关联，构建训练数据，形成大规模且低成本的数据集。
• 在自动标注和手动标注的数据上分别训练多种分类器——SVM、多项式朴素贝叶斯和随机森林，以进行对比。
• 还评估了集成分类器方法，以提升在自动标注数据上的性能。
• 在自建的情感阿拉伯语推文数据集上开展实验，以评估该方法的有效性。

实验结果

研究问题

• RQ1 基于表情符号的自动标注能否生成使情感分类器性能优于手动标注数据的训练数据？
• RQ2 在大规模自动标注数据上训练的情感分类器性能，与在小规模手动标注数据集上训练的分类器相比如何？
• RQ3 通过表情符号实现的远程监督是否能缓解情感分类任务中的标签噪声影响？
• RQ4 在自动标注数据上训练时，不同分类器（SVM、MNB、RF）及集成方法的性能表现如何？
• RQ5 在使用基于表情符号的弱监督训练情感分类器时，数据量在多大程度上超过标签质量成为决定性因素？

主要发现

• 在基于表情符号自动生成的标注数据上训练情感分类器，其性能优于在较小规模手动标注数据集上训练的模型。
• 自动标注数据集的规模约为手动标注数据集的一个数量级更大，但仍取得了更优的结果。
• 所有单个分类器（SVM、MNB、RF）以及集成模型在更大规模的自动标注数据上训练时表现更优。
• 结果表明，在这种弱监督设置下，数据量的增加所带来的优势超过了标签噪声带来的负面影响。
• 本研究证实，基于表情符号的远程监督是低资源和多语言环境下情感分类的一种可行且有效的方法。