[Paper Review] Sparse Neural Attentive Knowledge-based Models for Grade Prediction

Sara Morsy, George Karypis|arXiv (Cornell University)|Apr 22, 2019

Online Learning and Analytics19 references4 citations

TL;DR

This paper proposes Neural Attentive Knowledge-based Models (NAK) to improve grade prediction by learning dynamic, attention-based weights for prior courses instead of assuming equal contribution, as in CKRM. Using sparsemax activation, NAK achieves significantly higher accuracy than CKRM and other baselines, with sparse attention weights that identify key pre-requisites—both formal and informal—enabling better course planning.

ABSTRACT

Grade prediction for future courses not yet taken by students is important as it can help them and their advisers during the process of course selection as well as for designing personalized degree plans and modifying them based on their performance. One of the successful approaches for accurately predicting a student's grades in future courses is Cumulative Knowledge-based Regression Models (CKRM). CKRM learns shallow linear models that predict a student's grades as the similarity between his/her knowledge state and the target course. A student's knowledge state is built by linearly accumulating the learned provided knowledge components of the courses he/she has taken in the past, weighted by his/her grades in them. However, not all the prior courses contribute equally to the target course. In this paper, we propose a novel Neural Attentive Knowledge-based model (NAK) that learns the importance of each historical course in predicting the grade of a target course. Compared to CKRM and other competing approaches, our experiments on a large real-world dataset consisting of $\sim$1.5 grades show the effectiveness of the proposed NAK model in accurately predicting the students' grades. Moreover, the attention weights learned by the model can be helpful in better designing their degree plans.

Motivation & Objective

• To address the limitation of existing knowledge-based models like CKRM, which assume all prior courses contribute equally to future grade prediction.
• To improve prediction accuracy by learning the varying importance of each historical course for a target course.
• To leverage attention mechanisms in neural networks to automatically identify relevant pre-requisites, including informal or hidden ones.
• To enable better academic advising and personalized degree planning by interpreting learned attention weights.
• To evaluate the effectiveness of both soft and sparse attention mechanisms in the context of grade prediction using real-world university data.

Proposed method

• The NAK model uses a neural network to compute attention weights over prior courses, replacing the uniform weighting in CKRM with learned, dynamic weights.
• It employs both softmax and sparsemax activation functions to produce attention weights, with sparsemax promoting sparsity by setting irrelevant course weights to zero.
• The student's knowledge state is computed as a weighted sum of prior course embeddings, where weights are the learned attention scores.
• The model predicts a student's grade in a target course as the inner product between the student's attention-weighted knowledge state and the target course's embedding vector.
• The model is trained end-to-end on a large-scale dataset of ~1.5 million grades from a 16-year period at a major university.
• Row-centering is applied to grades to improve model performance by subtracting each student’s average prior grade.

Experimental results

Research questions

• RQ1 Can attention mechanisms improve grade prediction accuracy compared to traditional knowledge-based models like CKRM?
• RQ2 Can sparse attention (via sparsemax) better identify relevant pre-requisites by filtering out irrelevant prior courses?
• RQ3 Do the learned attention weights reflect both formal and informal pre-requisites that are not explicitly listed in course catalogs?
• RQ4 How does the performance of NAK compare to baseline models such as KRM(sum), KRM(avg), and matrix factorization in terms of RMSE and prediction accuracy?
• RQ5 Can the interpretability of attention weights support better academic advising and personalized degree planning?

Key findings

• NAK(sparse) achieved a ~2% lower RMSE than KRM(avg), the best-performing baseline, demonstrating significant improvement in prediction accuracy.
• NAK(sparse) achieved ~1% higher prediction accuracy within two tick errors compared to KRM(avg), indicating better precision in grade estimation.
• The sparsemax activation function successfully produced sparse attention weights, with only a few prior courses receiving non-zero attention, effectively filtering out irrelevant courses.
• The model correctly identified both formal pre-requisites (e.g., Intermediate German I for Intermediate German II) and informal ones (e.g., University Writing for German courses) based on learned attention weights.
• Qualitative analysis showed that attention weights were interpretable and aligned with academic requirements, even for non-major courses.
• The model’s ability to learn hidden pre-requisites without supervision suggests it captures underlying knowledge dependencies beyond explicit course catalogs.