[Paper Review] A Deep Learning Approach to Behavior-Based Learner Modeling

Yuwei Tu, Wei-Yu Chen+1 more|arXiv (Cornell University)|Jan 23, 2020

Online Learning and Analytics19 references4 citations

TL;DR

This paper proposes a Two Branch Decision Network (TBN) that combines clickstream behavior features and GloVe-based textual features from course content to predict learner performance in short online courses. The model achieves 95.7% accuracy and 0.958 AUC, outperforming both neural and non-neural baselines, demonstrating that integrating course content semantics with user behavior significantly improves prediction performance.

ABSTRACT

The increasing popularity of e-learning has created demand for improving online education through techniques such as predictive analytics and content recommendations. In this paper, we study learner outcome predictions, i.e., predictions of how they will perform at the end of a course. We propose a novel Two Branch Decision Network for performance prediction that incorporates two important factors: how learners progress through the course and how the content progresses through the course. We combine clickstream features which log every action the learner takes while learning, and textual features which are generated through pre-trained GloVe word embeddings. To assess the performance of our proposed network, we collect data from a short online course designed for corporate training and evaluate both neural network and non-neural network based algorithms on it. Our proposed algorithm achieves 95.7% accuracy and 0.958 AUC score, which outperforms all other models. The results also indicate the combination of behavior features and text features are more predictive than behavior features only and neural network models are powerful in capturing the joint relationship between user behavior and course content.

Motivation & Objective

• To improve learner outcome prediction in short online courses where intermediate assessments are absent.
• To investigate whether combining user behavior data with course content textual features enhances predictive performance.
• To develop a deep learning model that captures joint relationships between learner behavior and course content progression.
• To evaluate the model’s performance against both neural and non-neural network baselines on a real-world corporate training dataset.
• To explore the potential of using prediction results to inform actionable course content selection and personalization.

Proposed method

• The Two Branch Decision Network (TBN) processes user clickstream features and course content textual features through separate branches.
• Clickstream features are derived from user interactions (e.g., clicks, time-on-slide) with course content.
• Textual features are generated using pre-trained GloVe word embeddings to represent the semantic content of each slide and quiz.
• The two branches are fused via a fully connected layer to model the joint relationship between user behavior and content semantics.
• The model is trained using Adam optimization with a learning rate of 0.005, minimizing cross-entropy loss over 2000 epochs.
• A 5-fold stratified cross-validation strategy is used to ensure balanced class distribution during training and evaluation.

Experimental results

Research questions

• RQ1 Can combining user behavior features with course content textual features improve performance prediction accuracy in short online courses?
• RQ2 How does a deep neural network model that jointly models user behavior and content semantics compare to traditional models?
• RQ3 Does the inclusion of semantic content features lead to more stable and accurate predictions than behavior-only models?
• RQ4 Can the model identify key course segments based on prediction performance, enabling actionable insights for content selection?
• RQ5 Is the TBN architecture more effective than alternative neural network designs such as ESN or BNN in capturing user-content interactions?

Key findings

• The proposed Two Branch Decision Network (TBN) achieves 95.7% accuracy and 0.958 AUC, outperforming all other models tested, including a gradient boosting classifier.
• The TBN model significantly outperforms the BNN and ESN models, which achieve 93.2% and 94.2% accuracy, respectively, demonstrating the superiority of the fully connected fusion layer.
• Incorporating textual features from course content via GloVe embeddings improves prediction performance over behavior-only models, confirming the importance of content semantics.
• The gradient boosting classifier achieves the highest accuracy (97.6%) but has a lower AUC (0.911), indicating less robustness in class probability estimation.
• Training curves show that AUC and accuracy stabilize after 200 epochs, suggesting that early stopping could be used to prevent overfitting.
• The results indicate that neural network models are more powerful and stable than non-neural models in capturing complex relationships between user behavior and course content.