[论文解读] Prescribing Deep Attentive Score Prediction Attracts Improved Student Engagement

Youngnam Lee, Byung‐Soo Kim+5 more|arXiv (Cornell University)|Apr 27, 2020

Intelligent Tutoring Systems and Adaptive Learning参考文献 29被引用 4

一句话总结

本研究评估了一种深度注意力神经网络模型在智能辅导系统（Santa）中用于成绩预测的影响，表明更高的预测准确度能显著提升学生参与度。通过针对约78万用户的A/B测试，该模型将平均绝对误差从78.9（协同过滤）降低至49.8，使购买率提高15.19%，平均答题数从20.03增至22.73，总利润提升13%，证明精确的成绩预测能有效增强学习动机与主动参与度。

ABSTRACT

Intelligent Tutoring Systems (ITSs) have been developed to provide students with personalized learning experiences by adaptively generating learning paths optimized for each individual. Within the vast scope of ITS, score prediction stands out as an area of study that enables students to construct individually realistic goals based on their current position. Via the expected score provided by the ITS, a student can instantaneously compare one's expected score to one's actual score, which directly corresponds to the reliability that the ITS can instill. In other words, refining the precision of predicted scores strictly correlates to the level of confidence that a student may have with an ITS, which will evidently ensue improved student engagement. However, previous studies have solely concentrated on improving the performance of a prediction model, largely lacking focus on the benefits generated by its practical application. In this paper, we demonstrate that the accuracy of the score prediction model deployed in a real-world setting significantly impacts user engagement by providing empirical evidence. To that end, we apply a state-of-the-art deep attentive neural network-based score prediction model to Santa, a multi-platform English ITS with approximately 780K users in South Korea that exclusively focuses on the TOEIC (Test of English for International Communications) standardized examinations. We run a controlled A/B test on the ITS with two models, respectively based on collaborative filtering and deep attentive neural networks, to verify whether the more accurate model engenders any student engagement. The results conclude that the attentive model not only induces high student morale (e.g. higher diagnostic test completion ratio, number of questions answered, etc.) but also encourages active engagement (e.g. higher purchase rate, improved total profit, etc.) on Santa.

研究动机与目标

• 探究智能辅导系统（ITS）中成绩预测准确度的提升是否能增强学生参与度。
• 评估最先进的深度注意力神经网络模型与传统协同过滤方法在真实用户行为中的实际影响。
• 量化预测准确度与可衡量的参与度指标（如完成率、购买行为、收入）之间的因果关系。
• 为更高预测可靠性可增强用户对教育型AI系统信任与动机提供实证依据。

提出的方法

• 在韩国约78万用户使用的多平台TOEIC考试备考智能辅导系统Santa上开展受控A/B测试。
• 部署了两种成绩预测模型：一种基于协同过滤（MAE = 78.9），另一种基于深度注意力神经网络（MAE = 49.8）。
• 深度注意力模型采用预训练/微调策略：首先在答题正确性与响应及时性上进行预训练，随后使用基于Transformer的架构对考试成绩进行微调。
• 该模型利用注意力机制捕捉学生学习序列中的长程依赖关系，从而更准确地表征知识状态。
• 通过诊断测试完成率、会员注册率、答题数、购买率、ARPU及总利润等多个参与度指标监控用户行为。
• 所有财务指标均根据模型参数比例进行归一化处理，以确保公平比较。

实验结果

研究问题

• RQ1 更准确的成绩预测模型是否能提升智能辅导系统中学生的学习动机？
• RQ2 预测准确度的提升在多大程度上影响了用户的主动参与行为，如购买决策？
• RQ3 深度注意力神经网络模型是否能在真实世界参与度结果中超越协同过滤方法？
• RQ4 预测可靠性与用户留存率或收入生成之间是否存在可量化的因果关联？

主要发现

• 深度注意力模型的平均绝对误差（MAE）为49.8，显著低于协同过滤模型的78.9。
• 诊断测试完成率分别为65.90%（深度注意力模型）与64.93%（协同过滤模型），表明前者具有更高的学习动机。
• 会员注册率分别为44.55%（深度注意力模型）与43.13%（协同过滤模型）。
• 深度注意力模型用户在诊断测试后平均答题22.73道，高于协同过滤模型的20.03道。
• 深度注意力模型的购买率提升15.19%（2.73% vs. 2.37%），实现更高转化。
• 深度注意力模型总利润为162,933.88美元，协同过滤模型为142,949.55美元，归一化后提升13%。