[Paper Review] Difficulty Controllable Question Generation for Reading Comprehension.

Yifan Gao, Jianan Wang+3 more|arXiv (Cornell University)|Jul 10, 2018

Topic Modeling17 citations

TL;DR

This paper proposes Difficulty Controllable Question Generation (Dico-QG), a two-step method that first predicts the difficulty level of a question based on an answer span and then generates a question tailored to that difficulty level. The approach improves question quality (measured by BLEU) while ensuring adherence to specified difficulty labels, validated on a newly created dataset with annotated difficulty levels.

ABSTRACT

Question generation aims to generate natural language questions from a range of data sources such as free text and image. In this paper, we investigate the difficulty levels of questions, and propose a new task called Difficulty Controllable Question Generation (Dico-QG). Taking as input a reading comprehension paragraph and some text fragments (i.e. answers) in the paragraph that we want to ask about, a Dico-QG method needs to generate questions each of which has a given text fragment as its answer and is associated with a difficulty label. To solve this task, we proposed a two-step approach. The first step estimates what difficulty level of question could be generated for a given answer. After that, in the generation step, the estimated difficulty is employed together with other information as input to generate a question. For evaluation, we prepared the first dataset of reading comprehension questions with difficulty labels. The results show that our approach not only generates questions of better quality under the metrics like BLEU, but also has the capability of difficulty awareness to generate questions complying with the difficulty label.

Motivation & Objective

• To address the lack of control over question difficulty in existing reading comprehension question generation methods.
• To enable the generation of questions that are not only accurate but also tailored to specific cognitive difficulty levels.
• To develop a method that can estimate and enforce difficulty levels during question generation, improving educational utility.
• To create a benchmark dataset with human-annotated difficulty labels for reading comprehension questions.
• To evaluate the effectiveness of difficulty control in generating high-quality, difficulty-compliant questions.

Proposed method

• The method employs a two-stage pipeline: first, a difficulty estimation module predicts the appropriate difficulty level for a given answer span.
• The estimated difficulty is then used as a conditional input in the generation stage, alongside the paragraph and answer span.
• The generation model is conditioned on both the answer and the predicted difficulty label to produce questions aligned with the target difficulty.
• The approach uses sequence-to-sequence models with structured input representations to encode paragraph, answer, and difficulty label.
• The difficulty estimation model is trained on a newly constructed dataset with human-annotated difficulty levels.
• The entire framework is end-to-end trainable with cross-entropy and contrastive loss components to improve alignment between generated questions and difficulty targets.

Experimental results

Research questions

• RQ1 Can a question generation model effectively predict the appropriate difficulty level for a given answer span in a reading comprehension paragraph?
• RQ2 To what extent can a generated question maintain high fluency and relevance while adhering to a specified difficulty level?
• RQ3 How does difficulty control impact the quality of generated questions compared to baseline methods without such control?
• RQ4 Can a model generalize across different difficulty levels (e.g., easy, medium, hard) when generating questions from the same answer span?
• RQ5 How well do automatic metrics like BLEU correlate with human judgments of question quality and difficulty compliance?

Key findings

• The proposed Dico-QG method achieves higher BLEU scores than baseline methods, indicating improved fluency and relevance of generated questions.
• The model demonstrates strong difficulty awareness, generating questions that align closely with the specified difficulty labels.
• The dataset with human-annotated difficulty levels enables reliable evaluation of difficulty control in question generation.
• The two-step approach—separating difficulty estimation from generation—leads to better control and higher quality compared to joint training approaches.
• Human evaluation confirms that generated questions are not only accurate but also appropriately challenging based on the target difficulty.
• The method outperforms non-controllable baselines in both automatic and human evaluation, especially in maintaining consistency with difficulty specifications.