[Paper Review] Simplifying Paragraph-level Question Generation via Transformer Language Models

Luis Enrico Lopez, Diane Kathryn Cruz+2 more|arXiv (Cornell University)|May 3, 2020

Topic Modeling17 references4 citations

TL;DR

This paper proposes a simplified approach to paragraph-level question generation using only a single finetuned Transformer language model, without additional mechanisms, answer metadata, or complex architectures. Despite its simplicity, the model outperforms prior RNN-based Seq2Seq models by 8.62 and 14.27 points in METEOR and ROUGE_L scores, respectively, and matches state-of-the-art performance on SQuAD v1.1 while being significantly less complex.

ABSTRACT

Question generation (QG) is a natural language generation task where a model is trained to ask questions corresponding to some input text. Most recent approaches frame QG as a sequence-to-sequence problem and rely on additional features and mechanisms to increase performance; however, these often increase model complexity, and can rely on auxiliary data unavailable in practical use. A single Transformer-based unidirectional language model leveraging transfer learning can be used to produce high quality questions while disposing of additional task-specific complexity. Our QG model, finetuned from GPT-2 Small, outperforms several paragraph-level QG baselines on the SQuAD dataset by 0.95 METEOR points. Human evaluators rated questions as easy to answer, relevant to their context paragraph, and corresponding well to natural human speech. Also introduced is a new set of baseline scores on the RACE dataset, which has not previously been used for QG tasks. Further experimentation with varying model capacities and datasets with non-identification type questions is recommended in order to further verify the robustness of pretrained Transformer-based LMs as question generators.

Motivation & Objective

• To develop a simpler, more efficient question generation system that avoids complex model architectures and additional features.
• To evaluate whether a single finetuned Transformer language model can outperform more complex RNN-based Seq2Seq models in paragraph-level question generation.
• To investigate the impact of input formatting, context length, and answer-awareness on model performance.
• To identify failure modes and limitations of the proposed single-model approach.
• To provide a lightweight, faster, and more reproducible alternative to existing state-of-the-art QG systems.

Proposed method

• Fine-tuned GPT-2, a single decoder-only Transformer language model, on a reformatted version of the SQuAD v1.1 dataset for question generation.
• Reformatted SQuAD data into a continuous text sequence with [SEP] delimiters to mimic autoregressive language modeling.
• Used the OQPL (One Question Per Line) and AQPL (All Questions Per Line) formats to structure input-output pairs for training.
• Applied standard language modeling finetuning without additional mechanisms like attention masking, pointer networks, or answer tagging.
• Evaluated performance using standard NLP metrics: BLEU-4, METEOR, and ROUGE-L.
• Conducted ablation studies on answer-awareness, input formatting, and context length to isolate performance factors.

Experimental results

Research questions

• RQ1 Can a single finetuned Transformer language model generate high-quality questions without additional mechanisms like answer-awareness or attention masking?
• RQ2 How does input formatting (OQPL vs. AQPL) affect the quality and consistency of generated questions?
• RQ3 Does the length of the context paragraph significantly impact the model’s ability to generate accurate and relevant questions?
• RQ4 How does the inclusion of answer-awareness information affect performance when no explicit mechanism guides attention to the answer?
• RQ5 What are the primary failure modes of the model, and what factors contribute to its generation errors?

Key findings

• The proposed single-model approach outperforms prior RNN-based Seq2Seq models by 8.62 and 14.27 points in METEOR and ROUGE_L scores, respectively.
• The model achieves competitive performance on SQuAD v1.1, matching state-of-the-art results despite using only a single decoder-based Transformer and no answer-aware mechanisms.
• Answer-aware finetuning led to worse performance, particularly in BLEU-4 and marginally in ROUGE_L, suggesting that the model cannot effectively utilize answer-awareness without explicit architectural support.
• The model performs best with the OQPL format, indicating that separating questions improves training signal clarity and generation quality.
• Longer context paragraphs negatively affect generation quality, with performance degradation observed as context length increases.
• Failure modes include hallucination of answers, incorrect question structure, and generation of questions unrelated to the context, often due to attention misalignment or ambiguous context.