[Paper Review] Prescribing Deep Attentive Score Prediction Attracts Improved Student Engagement

Youngnam Lee, Byung‐Soo Kim+5 more|arXiv (Cornell University)|Apr 27, 2020

Intelligent Tutoring Systems and Adaptive Learning29 references4 citations

TL;DR

This study evaluates the impact of a deep attentive neural network model for score prediction in an intelligent tutoring system (Santa), demonstrating that higher prediction accuracy significantly improves student engagement. Using an A/B test with 780K users, the model reduced mean absolute error from 78.9 (collaborative filtering) to 49.8, resulting in 15.19% higher purchase rates, 22.73 vs. 20.03 average questions answered, and 13% higher total profit, proving that precise score prediction enhances both motivation and active engagement.

ABSTRACT

Intelligent Tutoring Systems (ITSs) have been developed to provide students with personalized learning experiences by adaptively generating learning paths optimized for each individual. Within the vast scope of ITS, score prediction stands out as an area of study that enables students to construct individually realistic goals based on their current position. Via the expected score provided by the ITS, a student can instantaneously compare one's expected score to one's actual score, which directly corresponds to the reliability that the ITS can instill. In other words, refining the precision of predicted scores strictly correlates to the level of confidence that a student may have with an ITS, which will evidently ensue improved student engagement. However, previous studies have solely concentrated on improving the performance of a prediction model, largely lacking focus on the benefits generated by its practical application. In this paper, we demonstrate that the accuracy of the score prediction model deployed in a real-world setting significantly impacts user engagement by providing empirical evidence. To that end, we apply a state-of-the-art deep attentive neural network-based score prediction model to Santa, a multi-platform English ITS with approximately 780K users in South Korea that exclusively focuses on the TOEIC (Test of English for International Communications) standardized examinations. We run a controlled A/B test on the ITS with two models, respectively based on collaborative filtering and deep attentive neural networks, to verify whether the more accurate model engenders any student engagement. The results conclude that the attentive model not only induces high student morale (e.g. higher diagnostic test completion ratio, number of questions answered, etc.) but also encourages active engagement (e.g. higher purchase rate, improved total profit, etc.) on Santa.

Motivation & Objective

• To investigate whether improved score prediction accuracy in an Intelligent Tutoring System (ITS) enhances student engagement.
• To evaluate the real-world impact of a state-of-the-art deep attentive neural network model versus traditional collaborative filtering on user behavior.
• To quantify the causal relationship between prediction accuracy and measurable engagement metrics such as completion rates, purchases, and revenue.
• To provide empirical evidence that higher prediction reliability increases user trust and motivation in educational AI systems.

Proposed method

• A controlled A/B test was conducted on Santa, a multi-platform ITS for TOEIC exam prep with ~780K users in South Korea.
• Two score prediction models were deployed: one based on collaborative filtering (MAE = 78.9) and another using a deep attentive neural network (MAE = 49.8).
• The deep attentive model employed a pre-training/fine-tuning scheme: first pre-trained on response correctness and timeliness, then fine-tuned for exam score prediction using a Transformer-based architecture.
• The model used attention mechanisms to capture long-range dependencies in student learning sequences, improving representation of knowledge states.
• User behavior was monitored across multiple engagement metrics: diagnostic test completion, membership registration, questions answered, purchase rate, ARPU, and total profit.
• All financial metrics were normalized based on model parameter ratios to ensure fair comparison.

Experimental results

Research questions

• RQ1 Does a more accurate score prediction model lead to higher student motivation in an ITS?
• RQ2 To what extent does improved prediction accuracy influence active user engagement, such as purchase decisions?
• RQ3 Can a deep attentive neural network model outperform collaborative filtering in real-world engagement outcomes?
• RQ4 Is there a measurable causal link between prediction reliability and user retention or revenue generation?

Key findings

• The deep attentive model achieved a mean absolute error (MAE) of 49.8, significantly lower than the collaborative filtering model’s MAE of 78.9.
• The diagnostic test completion rate was 65.90% for the deep attentive model versus 64.93% for the collaborative filtering model, indicating higher motivation.
• The registration (membership) rate was 44.55% for the deep attentive model, compared to 43.13% for the collaborative filtering model.
• Users of the deep attentive model answered an average of 22.73 questions after the diagnostic test, compared to 20.03 for the collaborative filtering model.
• The purchase rate increased by 15.19% with the deep attentive model (2.73%) versus the collaborative filtering model (2.37%).
• Total profit was $162,933.88 for the deep attentive model versus $142,949.55 for the collaborative filtering model, a 13% increase after normalization.