[Paper Review] A Self-Attentive model for Knowledge Tracing

Shalini Pandey, George Karypis|arXiv (Cornell University)|Jul 16, 2019

Intelligent Tutoring Systems and Adaptive Learning9 references189 citations

TL;DR

SAKT uses self-attention to identify relevant past exercises (KCs) for predicting a student's next response, outperforming state-of-the-art KT models and offering faster training than RNN-based approaches.

ABSTRACT

Knowledge tracing is the task of modeling each student's mastery of knowledge concepts (KCs) as (s)he engages with a sequence of learning activities. Each student's knowledge is modeled by estimating the performance of the student on the learning activities. It is an important research area for providing a personalized learning platform to students. In recent years, methods based on Recurrent Neural Networks (RNN) such as Deep Knowledge Tracing (DKT) and Dynamic Key-Value Memory Network (DKVMN) outperformed all the traditional methods because of their ability to capture complex representation of human learning. However, these methods face the issue of not generalizing well while dealing with sparse data which is the case with real-world data as students interact with few KCs. In order to address this issue, we develop an approach that identifies the KCs from the student's past activities that are extit{relevant} to the given KC and predicts his/her mastery based on the relatively few KCs that it picked. Since predictions are made based on relatively few past activities, it handles the data sparsity problem better than the methods based on RNN. For identifying the relevance between the KCs, we propose a self-attention based approach, Self Attentive Knowledge Tracing (SAKT). Extensive experimentation on a variety of real-world dataset shows that our model outperforms the state-of-the-art models for knowledge tracing, improving AUC by 4.43% on average.

Motivation & Objective

• Motivate KT as predicting student mastery over knowledge concepts from interaction sequences.
• Address data sparsity in KT by focusing on a few relevant past interactions rather than entire histories.
• Propose a self-attention based KT model that identifies relevant past exercises to predict future performance.
• Demonstrate that attention-driven relevance leads to better predictive performance and faster training than RNN-based KT models.

Proposed method

• Embed past interactions and exercises to form input representations.
• Apply positional encoding to capture sequence order.
• Use multi-head scaled dot-product self-attention to compute relevance between past interactions and the current exercise.
• Incorporate causality by masking future interactions in attention computations.
• Feed the attended representations through a feed-forward network with residual connections and layer normalization before predicting the next response with a sigmoid output.
• Train by minimizing binary cross-entropy loss on observed next responses.

Experimental results

Research questions

• RQ1 Can a self-attention KT model outperform state-of-the-art KT models (DKT, DKT+, DKVMN) across diverse datasets?
• RQ2 Does self-attention effectively identify and leverage the most relevant past exercises for predicting the next response, especially in sparse data settings?
• RQ3 Is the proposed architecture significantly faster to train than RNN-based KT models while maintaining or improving accuracy?
• RQ4 How interpretable are the attention weights in revealing relationships between exercises (KCs) and student performance?

Key findings

• SAKT outperforms state-of-the-art KT methods with an average AUC improvement of 4.43 percentage points across datasets.
• On synthetic data, SAKT achieves AUC of 0.832 versus 0.824 (DKT+), showing strong performance when concepts are clearly correlated within a single sequence.
• On ASSIST2009, SAKT achieves AUC of 0.848 versus 0.848 (DKT+ tied) and 0.820–0.829 for others, demonstrating improvement.
• On ASSIST2015, SAKT achieves AUC of 0.854, a substantial gain over baselines (e.g., 0.736–0.737 in competitors).
• On ASSISTChall and STATICS, SAKT matches or exceeds baseline methods, with notable gains on STATICS (0.853 vs 0.814–0.835).
• SAKT training is significantly faster than RNN-based models (e.g., 1.4s per epoch on a Titan V GPU versus 65s for DKT+ and 45s for DKT).
• Attention visualizations reveal clustering of latent concepts and relevance between exercises, illustrating interpretable relationships learned by the model.