[Paper Review] Assessing the Performance of Online Students -- New Data, New Approaches, Improved Accuracy

Robin Schmucker, Jingbo Wang+2 more|arXiv (Cornell University)|Sep 4, 2021

Online Learning and Analytics4 citations

TL;DR

This paper proposes a new machine learning approach for student performance modeling in online education, leveraging rich log data, curriculum prerequisite structures, and specialized models for different learning phases. By introducing novel features—such as response patterns, hint usage, and smoothed correctness rates—it achieves a state-of-the-art AUC of 0.808 across four large datasets, outperforming both traditional logistic regression and deep learning methods.

ABSTRACT

We consider the problem of assessing the changing performance levels of individual students as they go through online courses. This student performance (SP) modeling problem is a critical step for building adaptive online teaching systems. Specifically, we conduct a study of how to utilize various types and large amounts of student log data to train accurate machine learning (ML) models that predict the performance of future students. This study is the first to use four very large sets of student data made available recently from four distinct intelligent tutoring systems. Our results include a new ML approach that defines a new state of the art for logistic regression based SP modeling, improving over earlier methods in several ways: First, we achieve improved accuracy by introducing new features that can be easily computed from conventional question-response logs (e.g., the pattern in the student 's most recent answers). Second, we take advantage of features of the student history that go beyond question-response pairs (e.g., features such as which video segments the student watched, or skipped) as well as information about prerequisite structure in the curriculum. Third, we train multiple specialized SP models for different aspects of the curriculum (e.g., specializing in early versus later segments of the student history), then combine these specialized models to create a group prediction of the SP. Taken together, these innovations yield an average AUC score across these four datasets of 0.808 compared to the previous best logistic regression approach score of 0.767, and also outperforming state-of-the-art deep neural net approaches. Importantly, we observe consistent improvements from each of our three methodological innovations, in each dataset, suggesting that our methods are of general utility and likely to produce improvements for other online tutoring systems as well.

Motivation & Objective

• To improve the accuracy of student performance modeling in online learning environments using large-scale log data.
• To investigate how diverse student interaction features—beyond basic right/wrong responses—can enhance predictive modeling.
• To evaluate the impact of curriculum prerequisite structures and specialized models for different learning stages on prediction accuracy.
• To compare the performance of logistic regression and deep learning models using recent, large-scale ITS datasets.
• To establish a generalizable framework for student performance modeling applicable across diverse online tutoring systems.

Proposed method

• The method introduces new features derived from student log data, including response patterns (n-grams of recent correctness), hint usage (count and time), and smoothed average correctness rates.
• It incorporates curriculum background information, such as prerequisite and post-requisite relationships between knowledge components, to inform model predictions.
• The approach trains multiple specialized logistic regression models for distinct phases of student learning (e.g., early vs. late in the lesson) and combines their outputs for final prediction.
• Features like hint count and time are transformed using the log-scaled function φ(x) = log(1 + x) to reduce magnitude bias.
• A smoothed average correctness feature rs = (cs + ηr̄)/(as + η) is used with η = 5 to stabilize early student performance estimates.
• The study evaluates both logistic regression and deep learning models (DKT, SAKT, SAINT, SAINT+) using grid search over hyperparameters, with training over 100 epochs.

Experimental results

Research questions

• RQ1 How do different types of student interaction features (e.g., response patterns, hint usage, time on tasks) affect performance prediction accuracy?
• RQ2 To what extent does incorporating curriculum prerequisite structure improve student performance modeling?
• RQ3 Can training specialized models for different phases of student learning (e.g., initial vs. review stages) lead to better overall prediction performance?
• RQ4 How does the proposed method compare in accuracy to existing state-of-the-art approaches, including deep learning models?
• RQ5 Are the improvements observed consistent across diverse online tutoring systems and datasets?

Key findings

• The proposed method achieves an average AUC of 0.808 across four large, diverse datasets, significantly improving upon the previous best logistic regression approach (AUC 0.767).
• Each of the three core innovations—new features, curriculum structure integration, and specialized models—produced consistent accuracy gains across all four datasets.
• The method outperforms state-of-the-art deep learning models (DKT, SAKT, SAINT, SAINT+) in terms of AUC, demonstrating the effectiveness of structured logistic regression with rich features.
• The smoothed average correctness feature with η = 5 was found to reduce variance in early student performance estimation and improved model stability.
• Response pattern features enabled the model to detect short-term performance trends, such as streaks of correct/incorrect answers, enhancing predictive power.
• The use of log-scaled hint features (count and time) improved model robustness by mitigating the impact of extreme values in student behavior.