[Paper Review] A Generative Approach to Question Answering

Rajarshee Mitra|arXiv (Cornell University)|Nov 16, 2017

Topic Modeling7 references3 citations

TL;DR

This paper proposes a generative question answering model that directly generates abstractive answers from passage-question pairs using a sequence-to-sequence framework enhanced with a pointer-generator network and coverage mechanism. By learning to copy key entities from the passage and track attended positions to reduce repetition, the model achieves state-of-the-art ROUGE-L scores on MS-MARCO, outperforming extractive and baseline generative models while improving answer readability and correctness.

ABSTRACT

Question Answering has come a long way from answer sentence selection, relational QA to reading and comprehension. We shift our attention to generative question answering (gQA) by which we facilitate machine to read passages and answer questions by learning to generate the answers. We frame the problem as a generative task where the encoder being a network that models the relationship between question and passage and encoding them to a vector thus facilitating the decoder to directly form an abstraction of the answer. Not being able to retain facts and making repetitions are common mistakes that affect the overall legibility of answers. To counter these issues, we employ copying mechanism and maintenance of coverage vector in our model respectively. Our results on MS-MARCO demonstrate it's superiority over baselines and we also show qualitative examples where we improved in terms of correctness and readability

Motivation & Objective

• To develop a generative question answering model that produces abstractive answers without relying on extractive labels or span annotations.
• To address the challenge of factual inaccuracy and repetition in generated answers by incorporating a copying mechanism and coverage vector.
• To enable the model to dynamically switch between abstractive and extractive generation based on input data, eliminating dependency on external extraction modules.
• To improve answer readability and correctness by maintaining attention coverage and reducing redundant word generation.

Proposed method

• The model uses a bidirectional GRU encoder with shared weights to encode both the question and passage into contextual representations.
• It applies scaled multiplicative attention to compute alignment between question and passage representations, enabling query-aware encoding.
• A pointer-generator network computes a probability $ p_{\text{gen}} $ to decide whether to generate a word from the vocabulary or copy from the passage, improving factual consistency.
• A coverage vector is maintained as the cumulative sum of attention distributions to penalize over-attending to the same encoder states, reducing repetition.
• The decoder attends to the encoder states at each time step, with an additional coverage-aware term in the attention mechanism to guide diversity.
• The model is trained end-to-end using negative log-likelihood loss, with fixed GloVe embeddings and Adadelta optimization.

Experimental results

Research questions

• RQ1 Can a sequence-to-sequence model generate high-quality abstractive answers without requiring extractive span annotations or external labeling?
• RQ2 How effective is the pointer-generator mechanism in preserving factual entities while generating answers?
• RQ3 To what extent does the coverage mechanism reduce repetition in generated answers compared to standard seq2seq models?
• RQ4 Does the integration of coverage and copying improve both ROUGE-L scores and human-rated readability?
• RQ5 How does the model’s internal decision between copying and generating vary across different question types?

Key findings

• The proposed model achieves a ROUGE-L score of 74/59.5 on MS-MARCO development data, significantly outperforming the baseline seq2seq model (37.70 ROUGE-L).
• The model with both pointer-generator and coverage mechanisms achieves a perplexity of 4.35/4.62, showing better training stability and generation quality.
• The average $ p_{\text{gen}} $ value of 0.7 indicates that the model predominantly copies from the passage, reflecting the extractive nature of the MS-MARCO dataset.
• Qualitative analysis shows that the model retains core information and reduces repetition, improving answer readability compared to baselines.
• The coverage mechanism effectively curbs repeated word generation, as evidenced by reduced redundancy in generated answers.
• Self-attention did not improve performance, suggesting that the decoder’s attention mechanism already captures sufficient long-range dependencies.