[Paper Review] Transformer-based End-to-End Question Generation.

Luis Enrico Lopez, Diane Kathryn Cruz+2 more|arXiv (Cornell University)|May 3, 2020

Topic ModelingComputer Science14 references25 citations

TL;DR

This paper proposes a simple, single-model Transformer-based approach for end-to-end question generation that outperforms complex RNN-based Seq2Seq models without requiring answer metadata or additional mechanisms. Fine-tuning a single pretrained language model achieves state-of-the-art results, with gains of 8.62 and 14.27 in METEOR and ROUGE_L scores, respectively.

ABSTRACT

Question Generation (QG) is an important task in Natural Language Processing (NLP) that involves generating questions automatically when given a context paragraph. While many techniques exist for the task of QG, they employ complex model architectures, extensive features, and additional mechanisms to boost model performance. In this work, we show that transformer-based finetuning techniques can be used to create robust question generation systems using only a single pretrained language model, without the use of additional mechanisms, answer metadata, and extensive features. Our best model outperforms previous more complex RNN-based Seq2Seq models, with an 8.62 and a 14.27 increase in METEOR and ROUGE_L scores, respectively. We show that it also performs on par with Seq2Seq models that employ answer-awareness and other special mechanisms, despite being only a single-model system. We analyze how various factors affect the model's performance, such as input data formatting, the length of the context paragraphs, and the use of answer-awareness. In addition, we also look into the modes of failure that the model experiences and identify the reasons why it fails.

Motivation & Objective

• To develop a robust, end-to-end question generation system using only a single pretrained Transformer model.
• To eliminate the need for complex architectures, answer metadata, or extensive feature engineering.
• To evaluate whether a minimal, unified model can match or exceed the performance of more complex, specialized Seq2Seq models.
• To analyze the impact of input formatting, context length, and answer-awareness on model performance.
• To identify and diagnose failure modes in the generated questions.

Proposed method

• Fine-tuning a single pretrained Transformer model (e.g., BERT or similar) on a question generation task using standard encoder-decoder attention.
• Using only context paragraphs and corresponding answers as input, without explicit answer token marking or additional features.
• Employing standard sequence-to-sequence training with cross-entropy loss and autoregressive decoding.
• Applying input formatting techniques such as template-based input construction to guide the model.
• Evaluating performance using standard metrics: METEOR and ROUGE-L.
• Conducting ablation studies to assess the impact of context length and input formatting on generation quality.

Experimental results

Research questions

• RQ1 Can a single, fine-tuned Transformer model achieve competitive question generation performance without answer-awareness or auxiliary mechanisms?
• RQ2 How do input formatting and context length affect the quality of generated questions?
• RQ3 What are the primary failure modes in the generated questions, and what causes them?
• RQ4 How does the performance of the single-model Transformer compare to more complex RNN-based and answer-aware Seq2Seq models?
• RQ5 To what extent does answer-awareness improve performance in a minimal Transformer-based setup?

Key findings

• The proposed single-model Transformer approach outperforms previous RNN-based Seq2Seq models by 8.62 points in METEOR and 14.27 points in ROUGE_L.
• Despite not using answer-awareness or additional mechanisms, the model performs on par with more complex models that incorporate such features.
• Input formatting significantly influences model performance, with structured templates improving generation quality.
• Longer context paragraphs negatively impact generation quality, particularly in question relevance and fluency.
• Common failure modes include question repetition, incorrect question type generation, and hallucination of non-existent answers.
• The model struggles most with questions requiring coreference resolution or complex reasoning, even when answers are present in the context.