[Paper Review] I Do Not Understand What I Cannot Define: Automatic Question Generation With Pedagogically-Driven Content Selection

Tim Steuer, Anna Filighera+2 more|arXiv (Cornell University)|Oct 8, 2021

Topic Modeling45 references4 citations

TL;DR

This paper presents a novel automatic question generation (AQG) system for educational textbooks that selects content based on pedagogical principles, specifically targeting definitional statements to generate high-quality, comprehension-fostering questions. The method uses a pedagogically driven content selection mechanism grounded in reading comprehension theory, and evaluation by educational experts shows high linguistic quality and strong pedagogical relevance, indicating the system's potential to enhance deep text understanding.

ABSTRACT

Most learners fail to develop deep text comprehension when reading textbooks passively. Posing questions about what learners have read is a well-established way of fostering their text comprehension. However, many textbooks lack self-assessment questions because authoring them is timeconsuming and expensive. Automatic question generators may alleviate this scarcity by generating sound pedagogical questions. However, generating questions automatically poses linguistic and pedagogical challenges. What should we ask? And, how do we phrase the question automatically? We address those challenges with an automatic question generator grounded in learning theory. The paper introduces a novel pedagogically meaningful content selection mechanism to find question-worthy sentences and answers in arbitrary textbook contents. We conducted an empirical evaluation study with educational experts, annotating 150 generated questions in six different domains. Results indicate a high linguistic quality of the generated questions. Furthermore, the evaluation results imply that the majority of the generated questions inquire central information related to the given text and may foster text comprehension in specific learning scenarios.

Motivation & Objective

• To address the scarcity of self-assessment questions in higher education textbooks, which hinders deep text comprehension.
• To overcome the limitations of existing automatic question generation (AQG) systems that lack pedagogical grounding in content selection.
• To develop an AQG system that selects question-worthy content based on learning theory, particularly focusing on definitional statements as pedagogically valuable.
• To evaluate the linguistic and pedagogical quality of generated questions using educational experts in a diverse range of academic domains.
• To demonstrate that AQG systems can generate questions that target central propositions in text-bases, thereby supporting coherent mental model construction.

Proposed method

• The system identifies definitional statements in textbooks as primary candidates for question generation, based on the hypothesis that definitions represent core, pedagogically valuable information.
• It applies a content selection mechanism that prioritizes sentences containing explicit definitions, leveraging linguistic cues such as 'is', 'means', or 'refers to' to detect definitional structures.
• The question generation pipeline transforms selected definitional sentences into well-formed, answerable questions using template-based and neural generation techniques.
• The system accepts complete textbooks as input, enabling context-aware question generation without requiring pre-segmented text units.
• It integrates principles from the Construction-Integration (C-I) model of text comprehension, ensuring generated questions target central propositions in the text-base and situational model.
• An expert evaluation study was conducted with 150 generated questions across six domains to assess linguistic fluency and pedagogical relevance.

Experimental results

Research questions

• RQ1 Can an automatic question generation system produce linguistically fluent and answerable questions from arbitrary textbook content?
• RQ2 Does selecting content based on definitional statements lead to pedagogically meaningful questions that target central information in the text?
• RQ3 How do educational experts rate the linguistic and pedagogical quality of automatically generated questions in diverse academic domains?
• RQ4 To what extent do generated questions align with the central propositions of the text-base, supporting coherent mental model construction?
• RQ5 Can a theory-grounded content selection mechanism outperform heuristic or algorithmic approaches in AQG for educational contexts?

Key findings

• The generated questions exhibit high linguistic quality, with strong fluency and grammatical correctness as rated by educational experts.
• A majority of the generated questions were judged to be pedagogically appropriate, targeting central information in the text and supporting comprehension.
• Experts frequently disagreed on the usability of individual questions, indicating the subjective nature of pedagogical evaluation, though clear trends in quality were observed.
• The system successfully identified and transformed definitional statements into meaningful questions, suggesting that definitions are a reliable signal for question-worthy content.
• The evaluation results support the theoretical claim that targeting central propositions in the text-base enhances text comprehension, validating the pedagogical foundation of the approach.
• Despite limitations in inter-annotator agreement, the results indicate that the AQG system produces questions with measurable educational value in higher education contexts.