[Paper Review] Process-BERT: A Framework for Representation Learning on Educational Process Data

Alexander Scarlatos, Christopher G. Brinton+1 more|arXiv (Cornell University)|Apr 28, 2022

Online Learning and Analytics4 citations

TL;DR

This paper proposes Process-BERT, a BERT-style framework for pre-training and fine-tuning representations of educational process data, such as student clickstreams in online assessments. By leveraging self-supervised pre-training on sequential process data and transfer learning for downstream prediction tasks, the method achieves state-of-the-art performance on the NAEP 2019 competition dataset, demonstrating both predictive accuracy and meaningful behavioral clustering in learned representations.

ABSTRACT

Educational process data, i.e., logs of detailed student activities in computerized or online learning platforms, has the potential to offer deep insights into how students learn. One can use process data for many downstream tasks such as learning outcome prediction and automatically delivering personalized intervention. However, analyzing process data is challenging since the specific format of process data varies a lot depending on different learning/testing scenarios. In this paper, we propose a framework for learning representations of educational process data that is applicable across many different learning scenarios. Our framework consists of a pre-training step that uses BERT-type objectives to learn representations from sequential process data and a fine-tuning step that further adjusts these representations on downstream prediction tasks. We apply our framework to the 2019 nation's report card data mining competition dataset that consists of student problem-solving process data and detail the specific models we use in this scenario. We conduct both quantitative and qualitative experiments to show that our framework results in process data representations that are both predictive and informative.

Motivation & Objective

• To address the challenge of representing diverse, domain-specific educational process data across different learning scenarios.
• To develop a generic, transferable framework for representation learning on sequential educational process data, applicable beyond specific formats.
• To improve downstream prediction performance for learning outcomes using self-supervised pre-training followed by fine-tuning.
• To generate interpretable, meaningful representations that reflect actual student problem-solving behaviors and cognitive patterns.
• To evaluate the framework’s effectiveness on real-world educational data, specifically the NAEP 2019 assessment dataset.

Proposed method

• Pre-training uses a masked token prediction objective, similar to BERT, on sequential process data to learn contextual representations.
• The framework processes time-series educational process data, such as clickstreams, using a transformer-based architecture with positional encoding.
• A fine-tuning stage adapts the pre-trained representations to downstream supervised tasks like learning outcome prediction.
• The model employs attention mechanisms to dynamically weigh relevant parts of the sequence during representation extraction.
• Question-level and student-level representations are extracted from the final hidden states of a GRU or transformer encoder, enabling multi-level behavioral analysis.
• Clustering is applied to the learned representations to identify distinct behavioral patterns in student problem-solving sequences.

Experimental results

Research questions

• RQ1 Can a BERT-style pre-training framework effectively learn meaningful representations from diverse educational process data, such as problem-solving clickstreams?
• RQ2 Does the proposed transfer learning framework outperform existing methods in predicting student learning outcomes on the NAEP 2019 dataset?
• RQ3 Can the learned representations reveal interpretable behavioral patterns, such as task-switching or strategic problem-solving, in student process data?
• RQ4 How do representations at the question level compare to those at the student level in terms of clustering clarity and behavioral insight?
• RQ5 Can the framework be generalized to other types of educational process data beyond assessment clickstreams?

Key findings

• The Process-BERT framework achieved superior or competitive performance across multiple downstream learning outcome prediction tasks compared to existing methods on the NAEP 2019 dataset.
• Qualitative analysis revealed distinct clusters in question-level representations corresponding to different problem-solving strategies, such as direct solving, backtracking, and task-switching.
• Student-level representations also showed meaningful clustering, though less distinct than question-level clusters, indicating varied behavioral patterns over time.
• The model captured clear behavioral distinctions such as students who completed all questions versus those who ran out of time, with the latter group showing poor performance on subsequent blocks.
• Pre-training with BERT-style objectives significantly improved downstream prediction accuracy, demonstrating the value of self-supervised representation learning on process data.
• The framework’s ability to generate interpretable clusters suggests potential for use in identifying effective or ineffective learning strategies and informing personalized interventions.