[Paper Review] Attention-guided Generative Models for Extractive Question Answering

Peng Xu, Davis Liang+2 more|arXiv (Cornell University)|Oct 12, 2021

Topic Modeling29 references11 citations

TL;DR

This paper proposes a method to extract answer spans from pretrained generative question-answering models using cross-attention patterns, enabling extractive QA without additional parameters. By jointly training on generative and extractive objectives, the model achieves state-of-the-art performance on open-domain QA with fewer parameters and enables hallucination-free inference via attention-guided span fallback.

ABSTRACT

We propose a novel method for applying Transformer models to extractive question answering (QA) tasks. Recently, pretrained generative sequence-to-sequence (seq2seq) models have achieved great success in question answering. Contributing to the success of these models are internal attention mechanisms such as cross-attention. We propose a simple strategy to obtain an extractive answer span from the generative model by leveraging the decoder cross-attention patterns. Viewing cross-attention as an architectural prior, we apply joint training to further improve QA performance. Empirical results show that on open-domain question answering datasets like NaturalQuestions and TriviaQA, our method approaches state-of-the-art performance on both generative and extractive inference, all while using much fewer parameters. Furthermore, this strategy allows us to perform hallucination-free inference while conferring significant improvements to the model's ability to rerank relevant passages.

Motivation & Objective

• To leverage cross-attention patterns in pretrained generative models as a built-in architectural prior for extractive answer span prediction.
• To jointly train generative and extractive objectives to improve both extractive and generative QA performance.
• To reduce hallucination in generative QA by using cross-attention-aligned spans as a fallback mechanism.
• To use cross-attention scores as a proxy for passage relevance to enhance passage reranking.
• To improve interpretability of black-box generative QA systems by exposing attention-based evidence.

Proposed method

• Use decoder cross-attention weights between generated tokens and input context tokens to identify the most relevant context span for the answer.
• Apply a differentiable span extraction loss that encourages cross-attention to align with ground-truth answer spans during joint training.
• Train a single sequence-to-sequence Transformer model with a combined loss: standard cross-entropy for generation and span extraction loss based on attention alignment.
• Use the top-k attention-weighted context tokens as a fallback answer when the generative output is out-of-context or hallucinated.
• Compute passage scores using the sum of attention weights over answer-relevant tokens to rerank retrieved passages.
• Utilize the attention mechanism as a weak supervision signal for passage retrieval, improving reranking performance without additional training.

Experimental results

Research questions

• RQ1 Can cross-attention patterns in a generative QA model be used to extract exact answer spans without architectural modifications?
• RQ2 Does joint training on generative and extractive objectives improve performance on both extractive and generative QA tasks?
• RQ3 Can attention-based span extraction serve as an effective fallback to prevent hallucinations in generative QA?
• RQ4 Can cross-attention scores act as a reliable proxy for passage relevance in retrieval systems?
• RQ5 Does using attention patterns improve the interpretability of generative QA models in open-domain settings?

Key findings

• The proposed attention-guided span extraction method achieves extractive QA performance comparable to state-of-the-art extractive models while using significantly fewer parameters.
• On NaturalQuestions and TriviaQA, the model reaches state-of-the-art performance on both generative and extractive inference with a single, unified architecture.
• The method reduces hallucinations by using cross-attention-aligned spans as a fallback, improving reliability in industrial QA systems.
• Cross-attention scores serve as an effective proxy for passage relevance, improving passage reranking performance over prior neural retrieval baselines.
• Joint training with extractive supervision leads to a 5.5% absolute improvement in passage reranking accuracy on the FiD model, as shown in Table 4.
• The approach enhances model interpretability by exposing evidence-based attention patterns, making generative QA outputs more transparent and trustworthy.