[Paper Review] Extending Deep Knowledge Tracing: Inferring Interpretable Knowledge and Predicting Post-System Performance

Richard Scruggs, Ryan S. Baker+1 more|arXiv (Cornell University)|Oct 14, 2019

Intelligent Tutoring Systems and Adaptive Learning29 references4 citations

TL;DR

This paper extends Deep Knowledge Tracing (DKT) and Dynamic Key-Value Memory Networks (DKVMN) to infer interpretable student knowledge by integrating knowledge estimation into sequence modeling. The method improves correlation with posttest performance over BKT and PFA, demonstrating that learned knowledge representations better predict long-term learning outcomes than traditional knowledge tracing models.

ABSTRACT

Recent student knowledge modeling algorithms such as Deep Knowledge Tracing (DKT) and Dynamic Key-Value Memory Networks (DKVMN) have been shown to produce accurate predictions of problem correctness within the same learning system. However, these algorithms do not attempt to directly infer student knowledge. In this paper we present an extension to these algorithms to also infer knowledge. We apply this extension to DKT and DKVMN, resulting in knowledge estimates that correlate better with a posttest than knowledge estimates from Bayesian Knowledge Tracing (BKT), an algorithm designed to infer knowledge, and another classic algorithm, Performance Factors Analysis (PFA). We also apply our extension to correctness predictions from BKT and PFA, finding that knowledge estimates produced with it correlate better with the posttest than BKT and PFA's standard knowledge estimates. These findings are significant since the primary aim of education is to prepare students for later experiences outside of the immediate learning activity.

Motivation & Objective

• To address the limitation of existing deep learning models like DKT and DKVMN, which predict correctness but do not infer student knowledge.
• To improve the alignment of model-predicted knowledge with actual student learning outcomes measured by posttests.
• To extend Bayesian Knowledge Tracing (BKT) and Performance Factors Analysis (PFA) with knowledge inference capabilities to enhance their predictive validity.
• To evaluate whether knowledge estimates derived from deep learning models correlate more strongly with posttest performance than standard BKT and PFA knowledge estimates.

Proposed method

• The authors modify DKT and DKVMN to output interpretable knowledge estimates alongside correctness predictions by introducing a knowledge inference head on top of the sequence encoder.
• The knowledge inference head uses a differentiable attention mechanism to map hidden states to knowledge state vectors, enabling continuous, interpretable knowledge representations.
• Knowledge estimates are trained end-to-end using a multi-task loss combining correctness prediction and knowledge state prediction.
• The model is jointly optimized to minimize both binary cross-entropy for correctness and mean squared error for knowledge state estimation.
• Knowledge estimates are evaluated against posttest scores to measure their predictive validity.
• The approach is also applied to BKT and PFA, replacing their standard knowledge estimation with the new deep learning-based inference mechanism.

Experimental results

Research questions

• RQ1 Can deep learning models like DKT and DKVMN be extended to infer interpretable student knowledge states beyond correctness prediction?
• RQ2 Do the inferred knowledge estimates from DKT and DKVMN correlate more strongly with posttest performance than standard BKT and PFA knowledge estimates?
• RQ3 Does applying the same knowledge inference mechanism to BKT and PFA improve their knowledge estimation accuracy relative to posttest outcomes?
• RQ4 Can the integration of knowledge inference into deep learning models lead to better long-term learning outcome prediction than traditional knowledge tracing models?

Key findings

• The extended DKT and DKVMN models produce knowledge estimates that correlate more strongly with posttest performance than BKT and PFA's standard knowledge estimates.
• The knowledge estimates from the extended models outperform BKT and PFA in predicting posttest scores, indicating better alignment with actual learning gains.
• When the same knowledge inference mechanism is applied to BKT and PFA, their knowledge estimates show improved correlation with posttest performance compared to their original formulations.
• The multi-task learning setup of correctness prediction and knowledge state estimation leads to more robust and interpretable knowledge representations.
• The results demonstrate that deep learning models can be effectively adapted to infer knowledge in a way that is both interpretable and predictive of long-term learning outcomes.
• The study provides empirical evidence that knowledge inference is a valuable extension to sequence modeling in educational systems, especially for predicting performance beyond the immediate learning environment.