[Paper Review] Automated Essay Scoring based on Two-Stage Learning

Jiawei Liu, Yang Xu+1 more|arXiv (Cornell University)|Jan 23, 2019

Adversarial Robustness in Machine Learning16 references45 citations

TL;DR

The paper proposes a Two-Stage Learning Framework (TSLF) that combines semantic, coherence, and prompt-relevance signals from deep models with handcrafted features, then uses XGBoost for final scoring, achieving strong robustness to adversarial inputs on ASAP.

ABSTRACT

Current state-of-art feature-engineered and end-to-end Automated Essay Score (AES) methods are proven to be unable to detect adversarial samples, e.g. the essays composed of permuted sentences and the prompt-irrelevant essays. Focusing on the problem, we develop a Two-Stage Learning Framework (TSLF) which integrates the advantages of both feature-engineered and end-to-end AES models. In experiments, we compare TSLF against a number of strong baselines, and the results demonstrate the effectiveness and robustness of our models. TSLF surpasses all the baselines on five-eighths of prompts and achieves new state-of-the-art average performance when without negative samples. After adding some adversarial essays to the original datasets, TSLF outperforms the feature-engineered and end-to-end baselines to a great extent, and shows great robustness.

Motivation & Objective

• Motivate improving AES by leveraging both handcrafted features and deep semantic representations.
• Detect adversarial AES inputs such as permuted paragraphs and prompt-irrelevant essays.
• Improve robustness and accuracy by fusing stage-one scores with feature-engineered features in a boosting model.

Proposed method

• Derive sentence embeddings with a pre-trained BERT model and compute sentence vectors by averaging penultimate-layer hidden states.
• Stage 1 computes three scores using LSTM-based encoders: semantic score Se, coherence score Ce, and prompt-relevant score Pe, each trained with MSE losses.
• Stage 2 concatenates Se, Ce, Pe with handcrafted features and feeds them into an XGBoost regression model to produce final scores.
• Introduce Grammar Error Correction (GEC) and Spell checking as part of the handcrafted feature pipeline.
• Normalize ASAP scores to (0,1) for training; rescale predictions to original ranges for testing.
• Train using Adam for stage-one components; apply early stopping for the boosting stage.

Experimental results

Research questions

• RQ1 Can integrating deep-encoded features with handcrafted features improve AES performance over purely end-to-end or purely feature-based methods?
• RQ2 Can coherence and prompt-relevance signals detect adversarial inputs such as permuted paragraphs and prompt-irrelevant essays?
• RQ3 Does combining Se, Ce, Pe with handcrafted features via a boosting model yield robust performance under adversarial conditions?

Key findings

• TSLF-ALL outperforms baselines on five of eight prompts and achieves the best average performance on ASAP without adversarial samples.
• TSLF-ALL remains more robust than baselines when adversarial samples are added, due to coherence and prompt-relevant signals.
• In ablations, the last hidden state in the LSTM-based scores performs better than the average hidden state.
• GEC-assisted grammar features and comprehensive handcrafted features enhance AES effectiveness beyond spell-check alone.
• With adversarial inputs, end-to-end and feature-based baselines fail to maintain performance, while TSLF-ALL shows strong robustness.