[Paper Review] Question Generation for Reading Comprehension Assessment by Modeling How and What to Ask

Bilal Ghanem, Lauren Lutz Coleman+3 more|arXiv (Cornell University)|Apr 6, 2022

Topic Modeling4 citations

TL;DR

This paper proposes HTA-WTA, a two-stage question generation model that first pre-trains on general question generation (How to Ask) and then fine-tunes for targeted reading comprehension skills (What to Ask). It introduces a novel dataset with nine story-based reading comprehension skills (SBRCS), including inferential and literal questions, and demonstrates that HTA-WTA outperforms strong baselines, especially in few-shot settings with only 10% of training data, achieving high-quality, skill-specific questions without answer supervision.

ABSTRACT

Reading is integral to everyday life, and yet learning to read is a struggle for many young learners. During lessons, teachers can use comprehension questions to increase engagement, test reading skills, and improve retention. Historically such questions were written by skilled teachers, but recently language models have been used to generate comprehension questions. However, many existing Question Generation (QG) systems focus on generating literal questions from the text, and have no way to control the type of the generated question. In this paper, we study QG for reading comprehension where inferential questions are critical and extractive techniques cannot be used. We propose a two-step model (HTA-WTA) that takes advantage of previous datasets, and can generate questions for a specific targeted comprehension skill. We propose a new reading comprehension dataset that contains questions annotated with story-based reading comprehension skills (SBRCS), allowing for a more complete reader assessment. Across several experiments, our results show that HTA-WTA outperforms multiple strong baselines on this new dataset. We show that the HTA-WTA model tests for strong SCRS by asking deep inferential questions.

Motivation & Objective

• To address the lack of controlled, skill-targeted question generation in reading comprehension, especially for inferential questions.
• To create a new, comprehensive dataset of story-based reading comprehension skills (SBRCS) with both literal and inferential questions.
• To develop a two-stage question generation model (HTA-WTA) that improves generalizability and skill-specific question quality.
• To evaluate the model’s performance in low-resource (few-shot) settings, simulating real-world data scarcity.
• To enable automated, high-quality question generation for educational platforms without answer supervision.

Proposed method

• The HTA-WTA model uses a two-stage training process: first pre-training on a general question generation dataset (How to Ask), then fine-tuning on a skill-specific dataset (What to Ask).
• The model leverages transfer learning from existing QG datasets to improve generalization and reduce data requirements.
• A skill name token is injected into the input prompt to control the type of comprehension skill targeted in the generated question.
• The method is trained end-to-end without answer supervision, relying solely on passage and skill label input.
• The approach uses a sequence-to-sequence transformer architecture, with fine-tuning on a newly curated dataset of story-based passages and skill-annotated questions.
• Stratified sampling is used to simulate few-shot learning by training on only 10% of the dataset instances per skill.

Experimental results

Research questions

• RQ1 Can a two-stage question generation model effectively generate questions targeting specific reading comprehension skills without answer supervision?
• RQ2 How does the HTA-WTA model perform compared to strong baselines on a diverse, multi-skill reading comprehension dataset?
• RQ3 To what extent can the HTA-WTA model generalize when fine-tuned on only a small fraction of the dataset (e.g., 10%)?
• RQ4 Does the inclusion of skill name tokens improve the model’s ability to generate questions aligned with specific comprehension skills?
• RQ5 Can the proposed model generate high-quality inferential questions, which are more challenging than literal ones?

Key findings

• HTA-WTA outperforms multiple strong baselines on the new SBRCS dataset, demonstrating superior performance in generating skill-targeted questions.
• The model achieved a BLEURT score of 33.21 in the few-shot setting with only 10% of the training data, exceeding most baselines and even outperforming models trained on the full dataset.
• The model achieved perfect F1 scores (1.0) on the low-resource skills of Predicting and Figurative Language, indicating strong generalization on challenging, inferential question types.
• The use of skill name tokens significantly improved performance on underrepresented skills, suggesting effective disambiguation and control over question type.
• Performance showed a gradual improvement with increasing data, though a minor drop from 10% to 30% data was observed, consistent with prior findings on data sparsity effects.
• The model successfully generated high-quality, inferential questions without answer supervision, validating its utility in real-world educational applications.