[Paper Review] Structured Reordering for Modeling Latent Alignments in Sequence Transduction

Bailin Wang, Mirella Lapata+1 more|arXiv (Cornell University)|Jun 6, 2021

Natural Language Processing Techniques60 references14 citations

TL;DR

This paper proposes a neural sequence-to-sequence model that improves systematic generalization by modeling segment-level alignments as structured latent variables using separable permutations. By introducing a reordering-first, align-later framework with exact dynamic programming inference, the model enables end-to-end differentiable training and achieves state-of-the-art performance on semantic parsing and machine translation tasks, especially under distribution shift.

ABSTRACT

Despite success in many domains, neural models struggle in settings where train and test examples are drawn from different distributions. In particular, in contrast to humans, conventional sequence-to-sequence (seq2seq) models fail to generalize systematically, i.e., interpret sentences representing novel combinations of concepts (e.g., text segments) seen in training. Traditional grammar formalisms excel in such settings by implicitly encoding alignments between input and output segments, but are hard to scale and maintain. Instead of engineering a grammar, we directly model segment-to-segment alignments as discrete structured latent variables within a neural seq2seq model. To efficiently explore the large space of alignments, we introduce a reorder-first align-later framework whose central component is a neural reordering module producing {\it separable} permutations. We present an efficient dynamic programming algorithm performing exact marginal inference of separable permutations, and, thus, enabling end-to-end differentiable training of our model. The resulting seq2seq model exhibits better systematic generalization than standard models on synthetic problems and NLP tasks (i.e., semantic parsing and machine translation).

Motivation & Objective

• To address the failure of standard seq2seq models in out-of-distribution settings, particularly in systematic generalization to novel combinations of known concepts.
• To overcome the limitations of monotonic alignments in neural models by enabling non-monotonic, structured segment-level alignments.
• To maintain end-to-end differentiability while modeling complex, hierarchical alignments through latent reordering.
• To improve interpretability by making segment-to-segment alignments explicit through discrete latent variables.
• To enable better performance in low-resource and weakly-supervised settings through strong inductive bias from alignment modeling.

Proposed method

• Introduces a reordering-first, align-later framework where input sequences are reordered via a neural module producing separable permutations.
• Models separable permutations using hierarchical permutation trees to reflect linguistic structure and enable efficient inference.
• Develops an efficient dynamic programming algorithm for exact marginal and MAP inference over separable permutations.
• Applies continuous relaxation to the reordering module to allow end-to-end backpropagation and differentiable training.
• Integrates the reordering module with an existing monotonic alignment module to support non-monotonic, segment-level alignments.
• Uses the resulting model for sequence transduction tasks such as semantic parsing and machine translation with latent alignment supervision.

Experimental results

Research questions

• RQ1 Can modeling structured, non-monotonic segment-level alignments improve systematic generalization in neural sequence-to-sequence models?
• RQ2 How can separable permutations be efficiently inferred and trained end-to-end in a neural architecture?
• RQ3 Does a reordering-first approach with latent permutations outperform standard seq2seq and monotonic alignment models on OOD generalization tasks?
• RQ4 To what extent do latent alignments improve model interpretability in sequence generation?
• RQ5 Can this approach generalize to real-world NLP tasks like machine translation and semantic parsing under distribution shift?

Key findings

• The proposed ReMoto model, especially the Hard- ReMoto variant, achieves state-of-the-art BLEU scores on both EN-JA and ZH-EN translation tasks, outperforming standard seq2seq and baselines like SSNT and local reordering.
• On the LEN split of ZH-EN translation, ReMoto achieves 22.6 BLEU, significantly outperforming seq2seq (21.4) and SSNT (20.5), demonstrating improved OOD generalization.
• In semantic parsing, the model successfully generalizes to test examples composed of novel combinations of training segments, as shown in qualitative examples with explicit alignment rationales.
• The model's performance advantage is more pronounced in the LEN split (longer sentences), indicating better generalization under distribution shift.
• The use of separable permutations enables efficient exact inference via dynamic programming, making end-to-end training feasible despite the large search space.
• Latent alignments improve interpretability by explicitly linking output segments to input segments through reordered inputs, providing rationales for predictions.