[Paper Review] Question Generation from Paragraphs: A Tale of Two Hierarchical Models

Vishwajeet Kumar, Raktim Chaki+4 more|arXiv (Cornell University)|Nov 8, 2019

Topic Modeling19 references4 citations

TL;DR

This paper proposes two novel hierarchical models for paragraph-level question generation: a hierarchical BiLSTM with selective attention and a hierarchical Transformer architecture. Both models encode paragraphs hierarchically by processing words within sentences and sentences within paragraphs, leveraging attention mechanisms to improve relevance and fluency. The hierarchical Transformer outperforms all baselines in syntactic and semantic correctness, while the hierarchical BiLSTM achieves the highest relevance scores, demonstrating the effectiveness of hierarchical modeling for long-context question generation.

ABSTRACT

Automatic question generation from paragraphs is an important and challenging problem, particularly due to the long context from paragraphs. In this paper, we propose and study two hierarchical models for the task of question generation from paragraphs. Specifically, we propose (a) a novel hierarchical BiLSTM model with selective attention and (b) a novel hierarchical Transformer architecture, both of which learn hierarchical representations of paragraphs. We model a paragraph in terms of its constituent sentences, and a sentence in terms of its constituent words. While the introduction of the attention mechanism benefits the hierarchical BiLSTM model, the hierarchical Transformer, with its inherent attention and positional encoding mechanisms also performs better than flat transformer model. We conducted empirical evaluation on the widely used SQuAD and MS MARCO datasets using standard metrics. The results demonstrate the overall effectiveness of the hierarchical models over their flat counterparts. Qualitatively, our hierarchical models are able to generate fluent and relevant questions

Motivation & Objective

• To address the challenge of generating high-quality, relevant questions from long paragraph-level text, which remains underexplored compared to sentence-level question generation.
• To design hierarchical neural architectures that model paragraphs as sequences of sentences and sentences as sequences of words, capturing long-range dependencies more effectively.
• To evaluate the effectiveness of hierarchical attention mechanisms in improving question relevance and fluency compared to flat sequence models.
• To compare the performance of hierarchical BiLSTM and Transformer models on automatic and human evaluation metrics across SQuAD and MS MARCO datasets.

Proposed method

• The hierarchical BiLSTM model (HierSeq2Seq + AE) encodes words in each sentence using a BiLSTM, then encodes sentences into a paragraph-level representation using another BiLSTM, with word- and sentence-level selective attention to focus on salient content.
• The hierarchical Transformer model (HierTransSeq2Seq + AE) uses a multi-head attention mechanism at both word and sentence levels to build hierarchical representations, incorporating positional encoding for sequence order.
• Selective attention is applied at both word and sentence levels to dynamically attend to important tokens and sentences relevant to question generation.
• The models use an auto-encoding (AE) mechanism to improve training stability and generate more fluent questions.
• Both models are trained end-to-end using an encoder-decoder framework with attention, optimized for automatic metrics (BLEU, ROUGE) and human evaluation.
• The architecture is evaluated on SQuAD and MS MARCO datasets using standard automatic metrics and human annotation for syntax, semantics, and relevance.

Experimental results

Research questions

• RQ1 Does hierarchical modeling of paragraphs—processing sentences as units and words within sentences—improve question generation quality compared to flat sequence models?
• RQ2 How do selective attention mechanisms at word and sentence levels enhance the relevance and fluency of generated questions in hierarchical BiLSTM and Transformer models?
• RQ3 Does the hierarchical Transformer architecture outperform the hierarchical BiLSTM in generating syntactically and semantically correct questions?
• RQ4 To what extent do hierarchical models reduce the performance gap between automatic and human evaluation metrics in paragraph-level question generation?
• RQ5 What are the failure modes of hierarchical models, and how do they relate to long-context modeling and attention mechanisms?

Key findings

• The hierarchical BiLSTM model (HierSeq2Seq + AE) achieved the best performance on BLEU2–BLEU4 metrics on the SQuAD dataset and showed consistently strong results across all automatic metrics on the MS MARCO dataset.
• The hierarchical Transformer model (HierTransSeq2Seq + AE) outperformed all baselines in human evaluation for syntactic and semantic correctness on both SQuAD and MS MARCO datasets, with a syntax score of 86 and semantics score of 73.33 on SQuAD.
• The hierarchical BiLSTM model achieved the highest relevance score of 51.33 on SQuAD and 50.00 on MS MARCO in human evaluation, significantly outperforming other models in relevance to the input paragraph.
• On the MS MARCO dataset, the hierarchical BiLSTM models outperformed the hierarchical Transformer models across all automatic metrics, indicating a dataset-specific performance difference.
• The hierarchical models consistently outperformed their flat counterparts in both automatic and human evaluation, demonstrating that hierarchical representation learning is crucial for long-context question generation.
• Human evaluation revealed that the hierarchical Transformer model achieved the highest inter-rater agreement (kappa = 0.87) for syntax, indicating strong consensus on grammatical correctness.