[Paper Review] Reinforcement Learning Based Graph-to-Sequence Model for Natural Question Generation

Yu Chen, Lingfei Wu+1 more|arXiv (Cornell University)|Aug 14, 2019

Topic ModelingComputer Science75 references79 citations

TL;DR

The paper introduces an RL-based Graph2Seq model with a bidirectional graph neural encoder and a deep alignment network to generate natural questions from text, achieving state-of-the-art results on SQuAD.

ABSTRACT

Natural question generation (QG) aims to generate questions from a passage and an answer. Previous works on QG either (i) ignore the rich structure information hidden in text, (ii) solely rely on cross-entropy loss that leads to issues like exposure bias and inconsistency between train/test measurement, or (iii) fail to fully exploit the answer information. To address these limitations, in this paper, we propose a reinforcement learning (RL) based graph-to-sequence (Graph2Seq) model for QG. Our model consists of a Graph2Seq generator with a novel Bidirectional Gated Graph Neural Network based encoder to embed the passage, and a hybrid evaluator with a mixed objective combining both cross-entropy and RL losses to ensure the generation of syntactically and semantically valid text. We also introduce an effective Deep Alignment Network for incorporating the answer information into the passage at both the word and contextual levels. Our model is end-to-end trainable and achieves new state-of-the-art scores, outperforming existing methods by a significant margin on the standard SQuAD benchmark.

Motivation & Objective

• Leverage rich text structure by modeling passages as graphs to improve question generation quality.
• Integrate answer information effectively to guide question generation at multiple granularities.
• Mitigate training-inference mismatch via a hybrid objective combining cross-entropy and reinforcement learning losses.
• Encourage syntactic and semantic validity of generated questions through diverse reward signals.
• Evaluate on standard benchmarks to establish state-of-the-art performance and analyze component contributions.

Proposed method

• Propose a Graph2Seq generator with a Bidirectional Gated Graph Neural Network (BiGGNN) encoder.
• Construct passage graphs using static (syntax-based) and dynamic (semantics-aware) methods.
• Introduce a Deep Alignment Network (DAN) to fuse passage and answer information at word and contextual levels.
• Train a hybrid evaluator with a mixed objective of cross-entropy and reinforcement learning using SCST.
• Apply rewards from BLEU-4 and Word Mover’s Distance (WMD) to optimize semantic and syntactic quality.
• Evaluate with beam search and compare against multiple baselines on the SQuAD dataset.

Experimental results

Research questions

• RQ1 How can graph-structured representations of text improve natural question generation compared to sequence-based encoders?
• RQ2 Does incorporating answer information through a deep alignment mechanism improve question relevance and quality?
• RQ3 What is the impact of static syntax-based versus dynamic semantics-aware graph construction on QG performance?
• RQ4 Can a hybrid CE+RL objective outperform pure CE training for QG?
• RQ5 How do the proposed models compare to state-of-the-art baselines on SQuAD in automatic and human evaluations?

Key findings

• The proposed G2S sta +BERT+RL and G2S dyn +BERT+RL achieve new state-of-the-art scores on SQuAD test splits.
• Static graph construction generally outperforms dynamic graph construction across metrics.
• Incorporating answer information via DAN substantially improves BLEU-4 (and other metrics) over ablated models.
• Fine-tuning with REINFORCE further boosts performance across settings, especially with BERT.
• Human evaluation shows competitive syntactic and semantic quality compared to ground-truth and strong baselines.
• Ablation shows BiGGNN with forward/backward message passing and DAN are crucial for gains.