[Paper Review] Insertion Transformer: Flexible Sequence Generation via Insertion Operations

Mitchell Stern, William Chan+2 more|arXiv (Cornell University)|Feb 8, 2019

Natural Language Processing Techniques162 citations

TL;DR

The paper introduces the Insertion Transformer, a flexible sequence generator that inserts tokens at arbitrary positions, enabling both serial and parallel decoding with varied training orders and achieving competitive translation quality while using logarithmically many decoding iterations.

ABSTRACT

We present the Insertion Transformer, an iterative, partially autoregressive model for sequence generation based on insertion operations. Unlike typical autoregressive models which rely on a fixed, often left-to-right ordering of the output, our approach accommodates arbitrary orderings by allowing for tokens to be inserted anywhere in the sequence during decoding. This flexibility confers a number of advantages: for instance, not only can our model be trained to follow specific orderings such as left-to-right generation or a binary tree traversal, but it can also be trained to maximize entropy over all valid insertions for robustness. In addition, our model seamlessly accommodates both fully autoregressive generation (one insertion at a time) and partially autoregressive generation (simultaneous insertions at multiple locations). We validate our approach by analyzing its performance on the WMT 2014 English-German machine translation task under various settings for training and decoding. We find that the Insertion Transformer outperforms many prior non-autoregressive approaches to translation at comparable or better levels of parallelism, and successfully recovers the performance of the original Transformer while requiring only logarithmically many iterations during decoding.

Motivation & Objective

• Motivate and address limitations of fixed left-to-right autoregressive generation in sequence modeling.
• Propose an insertion-based iterative decoding framework that allows tokens to be inserted anywhere in the output canvas.
• Enable both fully autoregressive (one insertion at a time) and partially autoregressive (simultaneous insertions) decoding.
• Explore training orders (left-to-right, balanced binary tree, uniform/entropy-maximizing objectives) to improve robustness and efficiency.
• Demonstrate competitive BLEU results on WMT 2014 English-German with logarithmic decoding iterations and parallel decoding capabilities.

Proposed method

• Replace standard autoregressive decoding with insertion operations that add tokens at any position in the current canvas.
• Use a modified Transformer decoder that provides slot representations and attends to the full canvas via full decoder self-attention.
• Model the content-location distribution either jointly over (content, location) or in a factorized form (p(c|l), p(l)).
• Incorporate contextualized vocabulary bias and optional mixture-of-softmaxes to address large output spaces.
• Train with order-specific losses: left-to-right, balanced binary tree with center-focused weighting, or uniform/maximum-entropy loss; include termination handling (slot or sequence finalization) and an EOS penalty to control length.
• Inference supports greedy decoding and parallel decoding across slots when trained for slot finalization, achieving logarithmic iteration counts in favorable configurations.

Experimental results

Research questions

• RQ1 Can insertion-based generation match or exceed autoregressive and non-autoregressive translation quality while enabling efficient, parallel decoding?
• RQ2 How do different training orders (left-to-right, balanced binary tree, uniform entropy) and architectural variants affect performance and decoding efficiency?
• RQ3 Can the model grow output length dynamically without predicting fixed target length upfront?
• RQ4 What are the trade-offs between slot finalization and sequence finalization termination strategies for decoding and training?
• RQ5 To what extent can parallel insertion achieve sub-linear (log2 n) decoding iterations without sacrificing BLEU performance?

Key findings

• Insertion Transformer can match Transformer-level performance on WMT 2014 English-German while using logarithmically many decoding iterations.
• Parallel decoding with slot finalization achieves comparable or slightly better BLEU than greedy decoding, showing effective high-parallelism generation.
• Binary tree loss with distillation yields strong results, achieving BLEU of 25.80 on development with greedy decoding and 25.80–27.29 range depending on setup.
• EOS penalties significantly improve performance by preventing premature termination; tuned penalties can raise BLEU by up to ~4 points in some settings.
• Parallel decoding schemes approach the theoretical lower bound ⌊log2 n⌋+1 iterations and, in practice, require rarely more than about 10 iterations for typical sentence lengths.
• Different architectural variants (joint vs. conditional content-location modeling, contextualized vocabulary bias, mixture-of-softmaxes) provide modest gains over baselines, with gains diminishing after proper EOS tuning.