[Paper Review] Neural network approach to classifying alarming student responses to online assessment

Christopher M. Ormerod, Amy E. Harris|arXiv (Cornell University)|Sep 20, 2018

Online Learning and Analytics37 references4 citations

TL;DR

This paper proposes a deep learning approach using bidirectional stacked LSTM and GRU networks with attention mechanisms to detect alarming student responses in online assessments—such as self-harm, violence, or abuse—by classifying them from typical responses. The best-performing model, a bidirectional stacked LSTM with attention, achieved 98.7% alert detection at 10% review threshold, significantly improving early intervention in high-risk cases.

ABSTRACT

Automated scoring engines are increasingly being used to score the free-form text responses that students give to questions. Such engines are not designed to appropriately deal with responses that a human reader would find alarming such as those that indicate an intention to self-harm or harm others, responses that allude to drug abuse or sexual abuse or any response that would elicit concern for the student writing the response. Our neural network models have been designed to help identify these anomalous responses from a large collection of typical responses that students give. The responses identified by the neural network can be assessed for urgency, severity, and validity more quickly by a team of reviewers than otherwise possible. Given the anomalous nature of these types of responses, our goal is to maximize the chance of flagging these responses for review given the constraint that only a fixed percentage of responses can viably be assessed by a team of reviewers.

Motivation & Objective

• To develop an automated system that identifies alarming student responses in free-text online assessments, such as threats of self-harm or violence.
• To address the challenge of detecting subtle, context-dependent anomalous language in student responses that traditional automated scoring systems miss.
• To maximize alert detection rate under the constraint that only a fixed percentage of responses can be manually reviewed.
• To optimize the balance between sensitivity and false positives in high-stakes educational assessment environments.

Proposed method

• The study employs recurrent neural networks (RNNs), specifically GRUs and LSTMs, to model sequential text patterns in student responses.
• Models are trained using word embeddings and evaluated with 5-fold cross-validation on a held-out set of 1,000 alerts.
• The architecture includes stacked, bidirectional, and attention-augmented variants to improve context modeling and long-range dependency capture.
• Hyperparameter tuning focuses on recurrent unit size, stacking depth, bidirectionality, and attention mechanisms to maximize detection sensitivity.
• Performance is measured by the percentage of alerts correctly flagged across varying review thresholds (1% to 10%).
• Ensemble and aggregated model outputs are used to further improve detection performance beyond individual models.

Experimental results

Research questions

• RQ1 Which RNN architecture—GRU, LSTM, or their stacked, bidirectional, or attention-augmented variants—yields the highest alert detection performance in student responses?
• RQ2 How do architectural components like stacking, bidirectionality, and attention mechanisms individually impact detection sensitivity and false positive rates?
• RQ3 Can a deep learning model effectively distinguish contextually subtle alerts (e.g., hyperbolic expressions) from non-alarming responses with high precision?
• RQ4 To what extent can model performance be improved through ensemble averaging of multiple architectures?

Key findings

• The bidirectional stacked LSTM with attention mechanism achieved the highest alert detection rate of 98.7% when 10% of responses were reviewed.
• Stacking two layers of recurrent units (128,128) provided the largest performance gain, increasing detection by 14.98% compared to single-layer models.
• The use of attention mechanisms improved detection by 4.15% on average, while bidirectional modeling added 2.20%.
• The stacked LSTM model without attention achieved 96.7% detection at 10% review threshold, demonstrating strong baseline performance.
• The LSTM-based models outperformed GRU-based models by an average of 8.05% in detection accuracy.
• Aggregated model outputs further improved detection beyond the best individual model, suggesting potential for future ensemble-based optimization.