[论文解读] Creating A Neural Pedagogical Agent by Jointly Learning to Review and Assess

Youngnam Lee, Youngduck Choi+7 more|arXiv (Cornell University)|Jun 26, 2019

Topic Modeling参考文献 29被引用 15

一句话总结

本文提出了一种神经教学代理，通过在嵌入的问题-回答对上使用双向LSTM与注意力机制，实时联合学习对用户回答的评审与评估。该方法在预测回答正确性方面优于最先进方法，尤其在新用户上表现更优，同时通过学习到的注意力机制实现可解释的、基于标签的问题推荐，在拥有55.9万名用户的移动端英语学习平台上表现优异。

ABSTRACT

Machine learning plays an increasing role in intelligent tutoring systems as both the amount of data available and specialization among students grow. Nowadays, these systems are frequently deployed on mobile applications. Users on such mobile education platforms are dynamic, frequently being added, accessing the application with varying levels of focus, and changing while using the service. The education material itself, on the other hand, is often static and is an exhaustible resource whose use in tasks such as problem recommendation must be optimized. The ability to update user models with respect to educational material in real-time is thus essential; however, existing approaches require time-consuming re-training of user features whenever new data is added. In this paper, we introduce a neural pedagogical agent for real-time user modeling in the task of predicting user response correctness, a central task for mobile education applications. Our model, inspired by work in natural language processing on sequence modeling and machine translation, updates user features in real-time via bidirectional recurrent neural networks with an attention mechanism over embedded question-response pairs. We experiment on the mobile education application SantaTOEIC, which has 559k users, 66M response data points as well as a set of 10k study problems each expert-annotated with topic tags and gathered since 2016. Our model outperforms existing approaches over several metrics in predicting user response correctness, notably out-performing other methods on new users without large question-response histories. Additionally, our attention mechanism and annotated tag set allow us to create an interpretable education platform, with a smart review system that addresses the aforementioned issue of varied user attention and problem exhaustion.

研究动机与目标

• 解决移动端教育平台中用户行为动态变化、数据量庞大的实时用户建模挑战。
• 克服协同过滤与矩阵分解方法的局限性，后者在新数据到达时需要昂贵的重新训练。
• 开发一种系统，通过固定项目特征与动态更新的用户表征，高效建模用户知识状态。
• 通过利用主题标签与注意力机制，实现可解释的个性化问题推荐。

提出的方法

• 该模型使用双向循环神经网络（Bi-LSTM）编码用户回答与问题的序列，捕捉学习行为中的时间依赖性。
• 其在嵌入的问题-回答对上应用注意力机制，动态加权过去问题的相关性，以预测未来回答的正确性。
• 每次响应后实时更新用户特征，避免在数据摄入时进行完整模型重新训练。
• 该模型在大规模移动端教育数据集（SantaTOEIC）上端到端训练，包含6600万条回答记录与1万个带主题标注的问题。
• 注意力机制由专家标注的主题标签引导，实现基于语义相似性的可解释推荐。
• 通过标准指标（F1-score、AUC、准确率）评估模型架构，并对Bi-LSTM与注意力变体进行消融研究。

实验结果

研究问题

• RQ1 神经模型能否在不重新训练的情况下，实时联合学习评估与评审用户回答，从而提高预测准确性？
• RQ2 注意力机制如何通过与主题标签相似度对齐，提升问题推荐的可解释性？
• RQ3 与现有方法相比，该模型在响应历史较短的用户上是否仍保持优越性能？
• RQ4 双向LSTM与注意力机制在动态用户建模中的预测性能提升中分别起到何种贡献？
• RQ5 不同注意力机制（点积注意力 vs. 加性注意力）对模型性能与稳定性有何影响？

主要发现

• 所提模型取得0.8114的F1-score，优于所有基线方法，包括MF、MLP、NeuMF、RRN、AutoRec与DeepRec。
• 该模型在仅44次响应后即达到0.7的F1-score，而其他模型至少需要64次响应才能达到相似性能。
• 注意力机制与主题标签匹配比率呈现强相关性，前50名推荐问题的标签匹配比率达65–76%。
• 消融研究证实，尽管参数量相近，Bi-LSTM性能优于普通LSTM与全连接层。
• 注意力机制对性能有显著贡献，点积注意力与加性注意力表现近乎一致（F1: 0.8113–0.8114）。
• 该模型在响应历史极少的新用户上表现出强鲁棒性，凸显其在实时、可扩展的移动端教育平台中的适用性。