[论文解读] My Teacher Thinks The World Is Flat! Interpreting Automatic Essay Scoring Mechanism

Swapnil Parekh, Yaman Kumar+3 more|arXiv (Cornell University)|Dec 27, 2020

Topic Modeling参考文献 20被引用 7

一句话总结

本文通过使用积分梯度分析词级显著性，研究了基于深度学习的自动作文评分（AES）模型的可解释性。研究发现，AES模型将作文视为‘词的混合物’——仅一小部分高显著性词汇（如SkipFlow模型为31%，Memory Network模型为51%）决定评分，而无视上下文、词序或连贯性。这些模型容易受到对抗性扰动的影响，例如插入‘地球是平的’之类的虚假事实，由于过度依赖孤立关键词而非语义理解，此类扰动反而可能提高评分。

ABSTRACT

Significant progress has been made in deep-learning based Automatic Essay Scoring (AES) systems in the past two decades. However, little research has been put to understand and interpret the black-box nature of these deep-learning based scoring models. Recent work shows that automated scoring systems are prone to even common-sense adversarial samples. Their lack of natural language understanding capability raises questions on the models being actively used by millions of candidates for life-changing decisions. With scoring being a highly multi-modal task, it becomes imperative for scoring models to be validated and tested on all these modalities. We utilize recent advances in interpretability to find the extent to which features such as coherence, content and relevance are important for automated scoring mechanisms and why they are susceptible to adversarial samples. We find that the systems tested consider essays not as a piece of prose having the characteristics of natural flow of speech and grammatical structure, but as `word-soups' where a few words are much more important than the other words. Removing the context surrounding those few important words causes the prose to lose the flow of speech and grammar, however has little impact on the predicted score. We also find that since the models are not semantically grounded with world-knowledge and common sense, adding false facts such as ``the world is flat'' actually increases the score instead of decreasing it.

研究动机与目标

• 理解最先进深度学习-based AES 模型的黑箱评分机制。
• 探究为何这些模型会对语义不连贯或事实错误的作文给出高分。
• 评估连贯性、内容、相关性及事实准确性等语言特征在自动评分中的作用。
• 评估AES模型对保留词级重要性但破坏语义和结构的对抗性扰动的鲁棒性。
• 通过利用模型可解释性改进对抗性扰动技术，以实现更有效的测试。

提出的方法

• 作者使用积分梯度计算词级显著性图，识别对给定作文预测得分最具影响力的词汇。
• 通过移除或重排高显著性词汇并保持得分不变，评估模型鲁棒性，证明其对上下文和词序不敏感。
• 通过插入虚假事实陈述（如‘地球是平的’）进行对抗性测试，并测量得分变化。
• 比较两种最先进模型——SkipFlow（基于连贯性）和Memory Network（最先进）——在扰动下的得分稳定性。
• 分析基于显著性分数删除20%词汇与随机删除的影响，表明基于显著性的删除对预测得分影响极小。
• 考察词汇难度的影响，发现罕见或生僻词汇如‘legerdemain’和‘propinquity’反而被扣分。

实验结果

研究问题

• RQ1 基于深度学习的AES模型如何为作文中的单个词汇分配重要性？
• RQ2 为何AES模型会对包含‘地球是平的’等虚假事实的作文赋予高分？
• RQ3 模型预测在多大程度上依赖于词序、上下文或语义连贯性？
• RQ4 基于显著性的扰动与随机扰动相比，在保持预测得分方面表现如何？
• RQ5 在现代AES模型的评分机制中，罕见或生僻词汇起到何种作用？

主要发现

• SkipFlow模型仅使用由积分梯度识别出的31%最重要词汇，即可复现原始作文得分。
• 基于Memory Network的模型在保留51%最高显著性词汇时仍能维持原始得分，表明其严重依赖一小部分关键词。
• 移除高显著性词汇的上下文对预测得分影响极小，表明模型不理解语义或句法结构。
• 重新排列最高显著性词汇不会显著改变预测得分，证实了‘词的混合物’行为——词序无关紧要。
• 插入如‘地球是平的’等虚假事实陈述会提高预测得分，表明模型未基于世界知识或语义真实性进行判断。
• 罕见且低频的词汇如‘legerdemain’和‘propinquity’反而被扣分，与AES模型偏好生僻词汇的说法相矛盾。