[Paper Review] When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs

Marialena Bevilacqua, Kezia Oketch+6 more|arXiv (Cornell University)|Sep 25, 2023

Topic ModelingComputer Science3 citations

TL;DR

This study proposes a framework to evaluate how machine learning (ML)-based essay scoring models trained on human-written texts assess text generated by GPTs. Using 18,460 human and GPT-generated essays across six genres, it finds that transformer-based models like BERT score GPT-generated text 10–15% higher than human essays, despite being fine-tuned only on human data—likely due to pre-training data overlap. The work highlights risks in automated text assessment as generative AI becomes pervasive.

ABSTRACT

The use of machine learning (ML) models to assess and score textual data has become increasingly pervasive in an array of contexts including natural language processing, information retrieval, search and recommendation, and credibility assessment of online content. A significant disruption at the intersection of ML and text are text-generating large-language models such as generative pre-trained transformers (GPTs). We empirically assess the differences in how ML-based scoring models trained on human content assess the quality of content generated by humans versus GPTs. To do so, we propose an analysis framework that encompasses essay scoring ML-models, human and ML-generated essays, and a statistical model that parsimoniously considers the impact of type of respondent, prompt genre, and the ML model used for assessment model. A rich testbed is utilized that encompasses 18,460 human-generated and GPT-based essays. Results of our benchmark analysis reveal that transformer pretrained language models (PLMs) more accurately score human essay quality as compared to CNN/RNN and feature-based ML methods. Interestingly, we find that the transformer PLMs tend to score GPT-generated text 10-15\% higher on average, relative to human-authored documents. Conversely, traditional deep learning and feature-based ML models score human text considerably higher. Further analysis reveals that although the transformer PLMs are exclusively fine-tuned on human text, they more prominently attend to certain tokens appearing only in GPT-generated text, possibly due to familiarity/overlap in pre-training. Our framework and results have implications for text classification settings where automated scoring of text is likely to be disrupted by generative AI.

Motivation & Objective

• To investigate how ML-based automated essay scoring (AES) models trained on human-written essays assess text generated by GPTs.
• To analyze the moderating effects of essay genre and model architecture on scoring disparities between human and GPT-generated text.
• To identify linguistic cues and attention patterns that differentiate human and GPT-generated essays in the context of ML scoring.
• To develop a reproducible framework for studying the intersection of automated content generation and automated assessment in NLP.
• To provide empirical insights into the risks of automated scoring systems being biased toward AI-generated content, even when trained exclusively on human data.

Proposed method

• The study employs a rich testbed of 18,460 essays: 15,437 human-generated and 3,023 GPT-generated (1,537 GPT-3.5 and 1,486 GPT-4), using 68 prompts across six genres.
• It applies state-of-the-art ML models for AES, including BERT, RoBERTa, CNN/RNN, and feature-based models, to score both human and GPT-generated essays.
• A statistical model is used to jointly analyze the main effects and interactions of authorship (human vs. GPT), prompt genre, and scoring model architecture.
• Parallel representation analysis is conducted to compare linguistic features such as topics, sentiment, parts-of-speech, and spelling errors between human and GPT-generated texts.
• Attention weight analysis is performed on BERT-based models to examine which tokens the models attend to when scoring GPT vs. human essays.
• The framework, code, data, and prompt designs are made publicly available via GitHub to support reproducibility and future research.

Figure 1 : Overview of Research Design

Experimental results

Research questions

• RQ1 RQ1: How effective are state-of-the-art feature-based and deep learning models for automated essay scoring (AES)?
• RQ2 RQ2: How do AES models trained on human-generated content rate text generated by GPT models, and what is the moderating effect of document genre on such assessments?
• RQ3 RQ3: What linguistic categories and cues are most different between human and GPT-generated text, and how do these differences affect model attention and scoring?

Key findings

• Transformer-based models such as BERT and RoBERTa outperform CNN/RNN and feature-based models in scoring human essay quality, confirming their state-of-the-art performance in text classification.
• Despite being fine-tuned only on human essays, BERT-based models score GPT-3.5-generated text approximately 15% higher than human-written essays on average.
• In contrast, feature-based and CNN/RNN models score human-written essays 10–15% and 26–32% higher, respectively, than GPT-generated text, indicating a significant model-dependent bias.
• The disparity in scoring is linked to attention patterns in BERT models, which show increased attention to tokens such as proper nouns (e.g., author or character names) that appear in both pre-training data and GPT-generated text.
• Linguistic analysis reveals that differences between GPT and human essays in topics, sentiment, parts-of-speech, and spelling errors are more pronounced than cross-gender or cross-race differences in human user-generated content.
• The findings suggest that overlap in pre-training data between BERT/RoBERTa and GPT models may cause the former to favor GPT-generated text, even when the models are not explicitly trained on such content.

Figure 2 : Overview of Analysis Framework