[Paper Review] Improving Conditioning in Context-Aware Sequence to Sequence Models

Xinyi Wang, Jason Weston+2 more|arXiv (Cornell University)|Nov 21, 2019

Topic Modeling24 references9 citations

TL;DR

This paper proposes a novel sequence-to-sequence model architecture that improves conditioning on both a source query and a long context by interleaving source and context attention during decoding, while introducing a data augmentation method to enhance training. The approach achieves consistent improvements across three context-aware tasks—document-level machine translation, long-form question answering, and knowledge-grounded dialogue—by enabling better utilization of both input modalities, as validated by higher context usage and lower backward perplexity.

ABSTRACT

Neural sequence to sequence models are well established for applications which can be cast as mapping a single input sequence into a single output sequence. In this work, we focus on cases where generation is conditioned on both a short query and a long context, such as abstractive question answering or document-level translation. We modify the standard sequence-to-sequence approach to make better use of both the query and the context by expanding the conditioning mechanism to intertwine query and context attention. We also introduce a simple and efficient data augmentation method for the proposed model. Experiments on three different tasks show that both changes lead to consistent improvements.

Motivation & Objective

• To address the challenge of effective sequence-to-sequence generation when conditioned on both a short source query and a long context document.
• To improve model performance in context-aware tasks such as abstractive question answering, document-level machine translation, and knowledge-grounded dialogue.
• To design a model architecture that better integrates and utilizes information from both source and context sequences during decoding.
• To develop a data augmentation strategy that leverages the model's architecture to improve generalization and training efficiency.

Proposed method

• The model uses separate encoders for the source and context, preventing cross-attention between them to enforce modality separation.
• During decoding, the model alternates between attending to the source and the context via an interleaved attention mechanism, enabling dynamic interaction between both inputs.
• A soft localized attention window is applied in the context encoder to restrict long-range dependencies, acting as a regularizing inductive bias.
• A data augmentation technique is introduced that generates synthetic training examples by swapping or reordering context and source sequences, improving robustness and generalization.
• The model is trained end-to-end with standard cross-entropy loss, and evaluation uses metrics like backward perplexity and context usage percentage.
• The architecture is evaluated on three tasks: document-level NMT, long-form QA, and knowledge-grounded dialogue, using standard benchmarks.

Experimental results

Research questions

• RQ1 How can sequence-to-sequence models be improved to better condition on both a short source query and a long context document?
• RQ2 What architectural modifications enable more effective integration of source and context information during decoding?
• RQ3 How does interleaved attention between source and context compare to sequential or concatenated attention mechanisms in terms of performance and context utilization?
• RQ4 To what extent does the proposed data augmentation strategy improve model generalization and robustness across diverse context-aware tasks?
• RQ5 What is the impact of architectural constraints—such as localized attention and no cross-attention—on model performance and attention patterns?

Key findings

• The proposed interleaved attention model outperforms both the sequential and multitask baselines on all three tasks, achieving lower backward perplexity and higher context usage.
• On the ELI5 dataset, the model achieves a context usage percentage (U_ctx) of 82.3%, significantly outperforming the sequential baseline (69.1%) and the multitask baseline (74.5%).
• The model shows a 12.2% absolute improvement in context usage over the sequential baseline on the ELI5 dataset, indicating better utilization of relevant context information.
• Qualitative analysis confirms that the model generates more fluent and contextually grounded responses, such as correctly interpreting 'là' as 'then' in document-level translation using contextual cues.
• The attention analysis reveals that architectural constraints—like localized attention and no cross-attention—act as beneficial inductive biases, reducing reliance on long-range dependencies without harming performance.
• The data augmentation method improves model generalization, particularly in low-resource settings, by exposing the model to diverse source-context permutations during training.