[Paper Review] Towards Interpretable Deep Learning Models for Knowledge Tracing

Yu Lu, Deliang Wang+2 more|arXiv (Cornell University)|May 13, 2020

Intelligent Tutoring Systems and Adaptive Learning45 references4 citations

TL;DR

This paper proposes using layer-wise relevance propagation (LRP) to interpret predictions of RNN-based deep learning knowledge tracing (DLKT) models, enabling post-hoc explanation of model decisions. Experiments show LRP effectively identifies relevant questions and concept-level relationships, validating its feasibility for improving model interpretability in educational AI.

ABSTRACT

As an important technique for modeling the knowledge states of learners, the traditional knowledge tracing (KT) models have been widely used to support intelligent tutoring systems and MOOC platforms. Driven by the fast advancements of deep learning techniques, deep neural network has been recently adopted to design new KT models for achieving better prediction performance. However, the lack of interpretability of these models has painfully impeded their practical applications, as their outputs and working mechanisms suffer from the intransparent decision process and complex inner structures. We thus propose to adopt the post-hoc method to tackle the interpretability issue for deep learning based knowledge tracing (DLKT) models. Specifically, we focus on applying the layer-wise relevance propagation (LRP) method to interpret RNN-based DLKT model by backpropagating the relevance from the model's output layer to its input layer. The experiment results show the feasibility using the LRP method for interpreting the DLKT model's predictions, and partially validate the computed relevance scores from both question level and concept level. We believe it can be a solid step towards fully interpreting the DLKT models and promote their practical applications in the education domain.

Motivation & Objective

• Address the lack of interpretability in deep learning-based knowledge tracing (DLKT) models, which hinders practical adoption in education.
• Develop a post-hoc explanation method to make RNN-based DLKT models more transparent and understandable for educators and system designers.
• Validate the relevance scores generated by LRP at both question and concept levels to ensure they align with model predictions.
• Explore the internal concept-level relationships captured by the DLKT model through interpretability analysis.

Proposed method

• Apply layer-wise relevance propagation (LRP) to backpropagate relevance scores from the output layer to the input layer of an LSTM-based DLKT model.
• Design specific LRP rules for handling both weighted linear and multiplicative connections in the LSTM architecture.
• Use relevance scores to identify which past questions most influenced the model’s prediction on the next question.
• Aggregate concept-level relevance scores by grouping them according to the predicted concept, enabling visualization of concept dependencies.
• Perform question deletion experiments in decreasing/increasing order of relevance to test the predictive impact of high- and low-relevance questions.
• Visualize concept-level relationships using directed graphs, where edge weights represent LRP-derived relevance values.

Experimental results

Research questions

• RQ1 Can LRP effectively explain the prediction decisions of an RNN-based DLKT model at the question level?
• RQ2 Do the relevance scores computed by LRP correlate with the actual contribution of questions to model performance?
• RQ3 Can LRP uncover meaningful concept-level relationships that reflect prerequisite or prerequisite-like dependencies in knowledge structures?
• RQ4 Is the interpretability method robust enough to distinguish between correctly and falsely predicted sequences based on relevance patterns?

Key findings

• The LRP method successfully identifies relevant questions for model predictions, with over 90% of exercise sequences showing consistent prediction accuracy when questions are deleted in relevance order.
• Deleting high-relevance questions significantly reduces prediction accuracy on correctly predicted sequences, validating the reliability of LRP scores.
• Deleting low-relevance questions improves prediction accuracy on falsely predicted sequences, further confirming the validity of the relevance ranking.
• The concept-level visualization reveals meaningful clusters (e.g., algebra, geometry, statistics) and plausible prerequisite relationships, such as 'ordering real number' being influenced by 'division fraction' and 'square root'.
• The LRP-based interpretation method demonstrates feasibility in explaining both question-level and concept-level contributions, supporting its use in building interpretable DLKT models.
• The results suggest that LRP can partially restore the internal knowledge structure learned by the DLKT model, offering insights into its decision-making process.