[论文解读] Cooking Is All About People: Comment Classification On Cookery Channels Using BERT and Classification Models (Malayalam-English Mix-Code)

Subramaniam Kazhuparambil, Abhishek Kaushik|arXiv (Cornell University)|Jun 15, 2020

Sentiment Analysis and Opinion Mining被引用 8

一句话总结

本研究评估了传统机器学习模型与多语言Transformer模型（BERT、DistilBERT、XLM）在马拉雅拉姆语-英语混合编码的多语言YouTube评论分类中的表现。基于烹饪频道评论的自定义数据集，XLM取得了67.31%的最高准确率，优于随机森林（63.59%）等传统模型，表明尽管计算成本更高，基于Transformer的模型在低资源多语言文本分类任务中仍具有显著优势。

ABSTRACT

The scope of a lucrative career promoted by Google through its video distribution platform YouTube has attracted a large number of users to become content creators. An important aspect of this line of work is the feedback received in the form of comments which show how well the content is being received by the audience. However, volume of comments coupled with spam and limited tools for comment classification makes it virtually impossible for a creator to go through each and every comment and gather constructive feedback. Automatic classification of comments is a challenge even for established classification models, since comments are often of variable lengths riddled with slang, symbols and abbreviations. This is a greater challenge where comments are multilingual as the messages are often rife with the respective vernacular. In this work, we have evaluated top-performing classification models for classifying comments which are a mix of different combinations of English and Malayalam (only English, only Malayalam and Mix of English and Malayalam). The statistical analysis of results indicates that Multinomial Naive Bayes, K-Nearest Neighbors (KNN), Support Vector Machine (SVM), Random Forest and Decision Trees offer similar level of accuracy in comment classification. Further, we have also evaluated 3 multilingual transformer based language models (BERT, DISTILBERT and XLM) and compared their performance to the traditional machine learning classification techniques. XLM was the top-performing BERT model with an accuracy of 67.31. Random Forest with Term Frequency Vectorizer was the best performing model out of all the traditional classification models with an accuracy of 63.59.

研究动机与目标

• 为应对内容创作者因评论数量庞大及多语言、非正式内容导致的人工评论分析挑战。
• 评估传统机器学习模型与基于Transformer的语言模型在多语言（马拉雅拉姆语-英语）评论分类任务中的性能表现。
• 识别用于分类烹饪频道观众反馈的最有效模型配置。
• 评估在多种模型与特征工程技术下分类结果的统计显著性与鲁棒性。
• 为内容创作者提供一种实用且可扩展的工具，以自动提取多语言YouTube评论中的情感与反馈。

提出的方法

• 从马拉雅拉姆语烹饪频道收集并整理了1,000条YouTube评论，标注为三种情感类别：正面、负面与中性。
• 使用自定义的马拉雅拉姆语与英语停用词列表对评论进行预处理，由于语言结构复杂性，未对马拉雅拉姆语单词进行词干化处理。
• 应用四种向量化技术：TF-IDF、词袋计数向量器、TfidfVectorizer与二值化向量器，以提取传统模型的特征。
• 训练并评估了五种传统模型：多项式朴素贝叶斯、K近邻（KNN）、支持向量机（SVM）、随机森林与决策树。
• 对三种多语言Transformer模型（BERT、DistilBERT、XLM）进行微调，使用其未大小写敏感的base版本，在混合语言数据集上进行微调。
• 对所有模型进行超参数调优，并应用Friedman检验以评估性能差异的统计显著性。

实验结果

研究问题

• RQ1 对由马拉雅拉姆语-英语混合编码评论构成的多语言文本数据进行分类有多困难？
• RQ2 在不同特征工程技术组合下，哪种传统机器学习模型在多语言评论分类中表现最佳？
• RQ3 基于Transformer的模型（BERT、DistilBERT、XLM）在该多语言分类任务中的性能如何与传统模型相比较？
• RQ4 在多种分类模型之间，性能差异的统计显著性如何？
• RQ5 结合多种模型的混合方法是否能提升对非正式、多语言YouTube评论的分类准确率？

主要发现

• XLM在所有模型中取得了最高的整体准确率67.31%，优于其他所有模型，包括传统分类器。
• 在传统模型中，使用词频向量器的随机森林表现最佳，准确率为63.59%。
• 所有基于Transformer的模型均持续优于传统模型，XLM、DistilBERT与BERT的准确率均在65%-67%之间。
• Friedman检验确认了不同模型间性能差异具有统计显著性，排除了偶然因素的影响。
• 传统模型表现水平相近，表明模型选择本身的影响小于特征工程与模型架构的影响。
• 本研究揭示了一个权衡：尽管Transformer模型更准确，但其计算成本显著高于传统模型。