[Paper Review] Improving Controllability of Educational Question Generation by Keyword Provision

Ying-Hong Chan, Ho-Lam Chung+1 more|arXiv (Cornell University)|Dec 2, 2021

Topic Modeling14 references4 citations

TL;DR

This paper proposes a keyword-provision-based controllable question generation (CQG) framework, KPQG, to improve the controllability and diversity of educational question generation (QG). By incorporating user-provided keywords into the input, KPQG enables targeted, syntactically varied question generation, achieving a state-of-the-art BLEU-4 score of 20.19 on the RACE dataset using DeBERTa, significantly outperforming prior work.

ABSTRACT

Question Generation (QG) receives increasing research attention in NLP community. One motivation for QG is that QG significantly facilitates the preparation of educational reading practice and assessments. While the significant advancement of QG techniques was reported, current QG results are not ideal for educational reading practice assessment in terms of extit{controllability} and extit{question difficulty}. This paper reports our results toward the two issues. First, we report a state-of-the-art exam-like QG model by advancing the current best model from 11.96 to 20.19 (in terms of BLEU 4 score). Second, we propose to investigate a variant of QG setting by allowing users to provide keywords for guiding QG direction. We also present a simple but effective model toward the QG controllability task. Experiments are also performed and the results demonstrate the feasibility and potentials of improving QG diversity and controllability by the proposed keyword provision QG model.

Motivation & Objective

• To address the lack of controllability in existing educational QG models, where generated questions cannot be steered toward user expectations.
• To improve the difficulty and diversity of generated questions by moving beyond simple factoid questions toward more complex, exam-like questions.
• To explore a new QG setting—Controllable Question Generation (CQG)—where users provide keywords to guide question direction.
• To evaluate the effectiveness of pre-trained language models (PLMs) in enhancing QG quality for educational assessment.
• To demonstrate that keyword guidance enables syntactic control and improved alignment with desired question types.

Proposed method

• Proposes a new QG setting, CQG, where input includes passage, answer, and a set of keywords, extending the standard (passage, answer) pair.
• Develops KPQG, a simple yet effective model that integrates keywords into the input sequence using special tokens to guide generation.
• Employs pre-trained language models (DeBERTa, RoBERTa, BART) fine-tuned for question generation using an auto-regressive, masked-LM or seq2seq generation approach.
• Uses keyword insertion in the input prompt (e.g., [S]passage[S]answer[S]keywords[M]) to condition the generation process.
• Trains models on the EQG-RACE dataset, which is derived from the RACE exam dataset, to generate exam-like questions.
• Employs standard NLP metrics (BLEU, ROUGE-L, METEOR) to evaluate generation quality and controllability.

Experimental results

Research questions

• RQ1 Can keyword provision significantly improve the controllability of educational question generation models?
• RQ2 Does incorporating user-specified keywords lead to question outputs that are closer to desired, gold-standard questions?
• RQ3 Can keyword guidance enable control over syntactic structure, such as question type (e.g., 'who' vs. 'which')?
• RQ4 Does using pre-trained language models on exam-like datasets improve QG quality compared to prior methods?
• RQ5 Can the proposed method enhance both diversity and practical usability in educational assessment preparation?

Key findings

• The proposed KPQG model achieves a state-of-the-art BLEU-4 score of 20.19 on the EQG-RACE dataset, surpassing the previous SOTA of 11.96.
• Keyword-provided guidance enables the model to generate questions that are significantly closer to gold-standard questions than baseline models.
• With keywords like 'mars' or 'Megan Smith', KPQG successfully generates questions aligned with the intended focus, even when baseline models produce off-target questions.
• The model demonstrates syntactic control: using 'who' or 'which' as keywords results in corresponding question structures, enabling targeted syntactic variation.
• Case studies show that KPQG can generate diverse, contextually relevant questions by varying keyword inputs, enhancing practical utility for educators.
• The use of DeBERTa as a backbone model yields the best performance, indicating that strong pre-trained models are critical for high-quality educational QG.