[Paper Review] Delving Deeper into MOOC Student Dropout Prediction

Jacob Whitehill, Kiran Mohan+3 more|arXiv (Cornell University)|Feb 21, 2017

Online Learning and Analytics20 references50 citations

TL;DR

The paper evaluates how training/testing regimes affect MOOC dropout prediction accuracy, compares proxy-label in-situ training to post-hoc, and shows deeper neural nets can improve accuracy over logistic regression.

ABSTRACT

In order to obtain reliable accuracy estimates for automatic MOOC dropout predictors, it is important to train and test them in a manner consistent with how they will be used in practice. Yet most prior research on MOOC dropout prediction has measured test accuracy on the same course used for training the classifier, which can lead to overly optimistic accuracy estimates. In order to understand better how accuracy is affected by the training+testing regime, we compared the accuracy of a standard dropout prediction architecture (clickstream features + logistic regression) across 4 different training paradigms. Results suggest that (1) training and testing on the same course ("post-hoc") can overestimate accuracy by several percentage points; (2) dropout classifiers trained on proxy labels based on students' persistence are surprisingly competitive with post-hoc training (87.33% versus 90.20% AUC averaged over 8 weeks of 40 HarvardX MOOCs); and (3) classifier performance does not vary significantly with the academic discipline. Finally, we also research new dropout prediction architectures based on deep, fully-connected, feed-forward neural networks and find that (4) networks with as many as 5 hidden layers can statistically significantly increase test accuracy over that of logistic regression.

Motivation & Objective

• Assess how training/test regimes influence dropout prediction accuracy in MOOC settings.
• Compare post-hoc training, cross-course training, and in-situ proxy-label training using clickstream features.
• Evaluate whether training on many courses or on similar-field courses affects performance.
• Investigate whether deeper neural networks improve dropout prediction over logistic regression.
• Provide guidance for reliable accuracy estimation to support live MOOC interventions.

Proposed method

• Use clickstream features from 73 features (demographics and activity) to train predictors.
• Baseline models: demographic-only and last-interaction-day heuristics.
• Primary classifier: logistic regression with L2 regularization (equivalent to a 2-layer NN).
• Evaluate four training paradigms: post-hoc on same course, train on same-field course, train on many other courses, and in-situ proxy-label training.
• Normalize training/testing features appropriately for each paradigm (feature-wise z-score or percentile conversion for in-situ).
• Measure accuracy using AUC across weeks from launch toCERT eligibility (T0% to T100%).
• Experiment with deep fully-connected neural networks using Net2Net to compare depth/width against logistic regression.

Experimental results

Research questions

• RQ1 How does the choice of training paradigm affect MOOC dropout prediction accuracy in practice?
• RQ2 Can in-situ proxy-label training approach achieve competitive accuracy without using data from finished courses?
• RQ3 Do deeper neural architectures provide statistically significant gains over logistic regression for MOOC dropout prediction?
• RQ4 Is prediction accuracy robust across academic disciplines?
• RQ5 What are reliable practices for evaluating MOOC dropout predictors for real-world deployment?

Key findings

• Post-hoc training on the same course yields the highest average AUC of 90.20%.
• In-situ proxy-label training achieves 87.33% average AUC, close to post-hoc performance and enabling live deployment.
• Training on many other courses yields 85.56% average AUC, outperforming training on a single other-field course (76.85%).
• Baseline demographics predict poorly (58.85% AUC), while last-interaction-day baseline performs reasonably (82.45% AUC).
• Deeper neural networks (5 hidden layers, 5 neurons each) achieve 97.55% AUC, modestly beating logistic regression at 97.20% (p<0.00001).
• Deeper architectures require efficient training strategies (Net2Net) and can be trained rapidly on large MOOC datasets.