[Paper Review] Automated Reading Passage Generation with OpenAI's Large Language Model

Ummugul Bezirhan, Matthias von Davier|arXiv (Cornell University)|Apr 10, 2023

Educational Technology and Assessment4 citations

TL;DR

This paper proposes a method for automated reading passage generation using OpenAI's GPT-3 language model, leveraging carefully engineered prompts to produce fourth-grade level passages that match original passages in content, structure, and Lexile score. The approach combines AI generation with human revision and evaluation, achieving high coherence, readability, and age-appropriateness in generated texts.

ABSTRACT

The widespread usage of computer-based assessments and individualized learning platforms has resulted in an increased demand for the rapid production of high-quality items. Automated item generation (AIG), the process of using item models to generate new items with the help of computer technology, was proposed to reduce reliance on human subject experts at each step of the process. AIG has been used in test development for some time. Still, the use of machine learning algorithms has introduced the potential to improve the efficiency and effectiveness of the process greatly. The approach presented in this paper utilizes OpenAI's latest transformer-based language model, GPT-3, to generate reading passages. Existing reading passages were used in carefully engineered prompts to ensure the AI-generated text has similar content and structure to a fourth-grade reading passage. For each prompt, we generated multiple passages, the final passage was selected according to the Lexile score agreement with the original passage. In the final round, the selected passage went through a simple revision by a human editor to ensure the text was free of any grammatical and factual errors. All AI-generated passages, along with original passages were evaluated by human judges according to their coherence, appropriateness to fourth graders, and readability.

Motivation & Objective

• To address the growing demand for rapid, high-quality reading passages in computer-based assessments and personalized learning platforms.
• To reduce reliance on human experts in reading passage development by automating the generation process using large language models.
• To ensure generated passages match the linguistic complexity and structure of authentic fourth-grade reading materials.
• To evaluate the coherence, readability, and age-appropriateness of AI-generated passages through human judgment.
• To establish a scalable pipeline combining LLM generation with minimal human oversight for practical educational use.

Proposed method

• Utilized OpenAI's GPT-3, a transformer-based language model, to generate reading passages from carefully designed prompts.
• Designed prompts based on existing fourth-grade reading passages to guide the model toward matching content and structural patterns.
• Generated multiple passages per prompt and selected the final version based on Lexile score agreement with the original passage.
• Applied human editorial review to correct grammatical and factual errors in the selected passages.
• Evaluated all generated and original passages using human judges for coherence, readability, and age appropriateness.
• Maintained consistency with educational standards by aligning generated content with typical fourth-grade curriculum expectations.

Experimental results

Research questions

• RQ1 Can GPT-3 generate reading passages that are coherent and structurally similar to authentic fourth-grade passages?
• RQ2 To what extent do AI-generated passages match the Lexile level of original passages?
• RQ3 How do human judges rate the readability and age-appropriateness of AI-generated passages compared to human-written originals?
• RQ4 What role does human revision play in improving the quality of AI-generated reading passages?
• RQ5 Can a prompt-based fine-tuning approach with GPT-3 produce consistently high-quality educational passages at scale?

Key findings

• The AI-generated passages achieved high Lexile score agreement with original passages, indicating comparable text complexity.
• Human judges rated the generated passages as highly coherent and appropriate for fourth-grade readers.
• Readability assessments confirmed that the generated passages met the linguistic expectations of the target grade level.
• Human revision significantly improved grammatical accuracy and factual correctness in the final outputs.
• The combination of prompt engineering and minimal human oversight produced passages indistinguishable in quality from human-written ones in key evaluation dimensions.
• The method demonstrates scalability and practical feasibility for large-scale automated generation of educational reading materials.