[Paper Review] Automated Scoring for Reading Comprehension via In-context BERT Tuning

Nigel Fernandez, Aritra Ghosh+5 more|arXiv (Cornell University)|May 19, 2022

Topic Modeling4 citations

TL;DR

This paper proposes in-context BERT fine-tuning for automated scoring of reading comprehension responses, using a shared model that incorporates item-specific context via structured input prompts. It achieves state-of-the-art performance on the NAEP challenge by leveraging shared passage and question context across items, outperforming both LM-based and non-LM baselines while demonstrating modest demographic bias.

ABSTRACT

Automated scoring of open-ended student responses has the potential to significantly reduce human grader effort. Recent advances in automated scoring often leverage textual representations based on pre-trained language models such as BERT and GPT as input to scoring models. Most existing approaches train a separate model for each item/question, which is suitable for scenarios such as essay scoring where items can be quite different from one another. However, these approaches have two limitations: 1) they fail to leverage item linkage for scenarios such as reading comprehension where multiple items may share a reading passage; 2) they are not scalable since storing one model per item becomes difficult when models have a large number of parameters. In this paper, we report our (grand prize-winning) solution to the National Assessment of Education Progress (NAEP) automated scoring challenge for reading comprehension. Our approach, in-context BERT fine-tuning, produces a single shared scoring model for all items with a carefully-designed input structure to provide contextual information on each item. We demonstrate the effectiveness of our approach via local evaluations using the training dataset provided by the challenge. We also discuss the biases, common error types, and limitations of our approach.

Motivation & Objective

• To address the scalability and context-leveraging limitations of existing automated scoring models that train a separate model per item.
• To develop a unified, scalable scoring model for reading comprehension that leverages shared passage and question context across items.
• To improve scoring accuracy by designing a context-rich input format that enables in-context fine-tuning of BERT without requiring separate models per item.
• To evaluate the model’s performance, error patterns, and fairness across demographic groups in a real-world educational assessment setting.

Proposed method

• The method uses in-context fine-tuning of a pre-trained BERT model by constructing a prompt that includes the passage, question, and example responses with their scores for each item.
• A shared BERT model is fine-tuned on all items simultaneously, using a structured input format that embeds contextual information for each target response.
• The input format includes the question, a few in-context examples (with gold scores), and the student’s response, all concatenated into a single sequence for BERT.
• The model is trained to predict the human-graded score of the student response based on this context-rich input, enabling zero-shot generalization to new items.
• Demographic information (e.g., gender, ethnicity) is optionally included as a prefix prompt to study fairness, such as "score this answer written by a Female Asian student".
• A post-hoc spell-checking step is applied to improve accuracy, and attention maps are used to analyze model interpretability.

Experimental results

Research questions

• RQ1 Can a single shared BERT model achieve high scoring accuracy across multiple reading comprehension items without separate fine-tuning per item?
• RQ2 How effective is in-context fine-tuning with structured prompts in capturing item-specific context while sharing parameters across items?
• RQ3 What types of errors does the model commonly make, and how do they relate to linguistic features like grammar, subjectivity, or coreference?
• RQ4 To what extent does the model exhibit bias toward different student demographic groups, and does including demographic information in the input affect fairness?
• RQ5 Can the model generalize to new items without access to the full passage, and how does this impact its reading comprehension capability?

Key findings

• The proposed in-context BERT fine-tuning approach outperforms both existing LM-based and non-LM-based automated scoring baselines on the NAEP training dataset.
• The model achieves high accuracy without requiring separate models per item, significantly improving scalability and reducing storage demands.
• Common error types include spelling/grammar mistakes, subjective interpretation, rare correct answers, and imitation of incorrect responses, especially when responses mimic low-scoring examples.
• The model exhibits modest bias, overestimating scores for most demographic groups, with the exception of Pacific Islanders/American Indians in grade 4, likely due to low training data size.
• Including demographic information in the input prompt slightly alters bias patterns, but the model still shows unavoidable but small biases across groups.
• Preprocessing with spell-checking improves scoring accuracy, and fine-tuning BERT is significantly more effective than using it only as a fixed text encoder.