[Paper Review] Time-Series Insights into the Process of Passing or Failing Online University Courses Using Neural-Induced Interpretable Student States.

Byungsoo Jeon, Eyal Shafran+3 more|arXiv (Cornell University)|May 1, 2019

Online Learning and Analytics1 citations

TL;DR

This paper proposes a time-series model that integrates clickstream data and textual notes from human mentors to generate interpretable, evolving student state representations for predicting online course failure. By incorporating natural language signals from mentor notes, the model improves both prediction accuracy and interpretability of student engagement trajectories.

ABSTRACT

This paper addresses a key challenge in Educational Data Mining, namely to model student behavioral trajectories in order to provide a means for identifying students most at-risk, with the goal of providing supportive interventions. While many forms of data including clickstream data or data from sensors have been used extensively in time series models for such purposes, in this paper we explore the use of textual data, which is sometimes available in the records of students at large, online universities. We propose a time series model that constructs an evolving student state representation using both clickstream data and a signal extracted from the textual notes recorded by human mentors assigned to each student. We explore how the addition of this textual data improves both the predictive power of student states for the purpose of identifying students at risk for course failure as well as for providing interpretable insights about student course engagement processes.

Motivation & Objective

• To address the challenge of identifying at-risk students in online university courses using time-series modeling of behavioral data.
• To explore the potential of textual data—specifically mentor-recorded notes—as a complementary signal to clickstream data in student state modeling.
• To improve the predictive power of student state representations for course failure while enhancing interpretability of engagement patterns.
• To provide actionable, interpretable insights into student behavioral trajectories for timely academic interventions.

Proposed method

• The model constructs dynamic student state representations by fusing clickstream data and embeddings from textual notes written by human mentors.
• Textual notes are processed using NLP techniques to extract semantic signals, which are then embedded into a shared latent space with clickstream features.
• A recurrent neural network (RNN)-based architecture models the temporal evolution of student states over the course duration.
• Interpretable attention mechanisms are applied to highlight key behavioral and textual signals influencing predictions.
• The model is trained end-to-end to predict final course outcomes (pass/fail) while preserving interpretability of contributing factors.
• Model performance is evaluated using standard metrics such as AUC-ROC and precision-recall, with ablation studies to isolate the impact of textual data.

Experimental results

Research questions

• RQ1 How does incorporating mentor-written textual notes improve the predictive accuracy of student state models in online courses?
• RQ2 To what extent do textual signals from mentors enhance the interpretability of student engagement trajectories?
• RQ3 How do the combined signals of clickstream activity and textual notes compare to clickstream-only models in identifying at-risk students?
• RQ4 Which specific textual and behavioral patterns are most indicative of course failure or success?

Key findings

• The inclusion of textual notes from mentors significantly improved the AUC-ROC score of the student state model compared to clickstream-only baselines.
• The model achieved higher precision in identifying students who ultimately failed, particularly in early course stages.
• Interpretable attention mechanisms highlighted specific mentor notes and clickstream patterns that correlated strongly with student attrition.
• Textual signals provided meaningful context about student motivation and engagement, such as expressions of confusion or disengagement, that clickstream data alone could not capture.