[Paper Review] Levenshtein Transformer

Jiatao Gu, Changhan Wang+1 more|arXiv (Cornell University)|May 27, 2019

Natural Language Processing Techniques19 references186 citations

TL;DR

Levenshtein Transformer (LevT) combines insertion and deletion operations in a dual-policy, imitation-learning framework to enable flexible, partially autoregressive sequence generation and refinement with efficiency gains over standard transformers.

ABSTRACT

Modern neural sequence generation models are built to either generate tokens step-by-step from scratch or (iteratively) modify a sequence of tokens bounded by a fixed length. In this work, we develop Levenshtein Transformer, a new partially autoregressive model devised for more flexible and amenable sequence generation. Unlike previous approaches, the atomic operations of our model are insertion and deletion. The combination of them facilitates not only generation but also sequence refinement allowing dynamic length changes. We also propose a set of new training techniques dedicated at them, effectively exploiting one as the other's learning signal thanks to their complementary nature. Experiments applying the proposed model achieve comparable performance but much-improved efficiency on both generation (e.g. machine translation, text summarization) and refinement tasks (e.g. automatic post-editing). We further confirm the flexibility of our model by showing a Levenshtein Transformer trained by machine translation can straightforwardly be used for automatic post-editing.

Motivation & Objective

• Introduce a new sequence generation model based on insertion and deletion operations.
• Achieve competitive or better performance than Transformer baselines on MT and text summarization.
• Enable efficient decoding with parallelizable refinement steps leading to speed-ups.
• Unify generation and refinement in a single model to support post-editing and editing-style tasks.
• Propose a dual-policy imitation learning algorithm exploiting the complementary nature of insertion and deletion.

Proposed method

• Model uses a Transformer backbone with three heads: deletion, placeholder insertion, and token insertion, sharing weights by default.
• Sequence editing proceeds in iterations: delete tokens, insert placeholders, replace placeholders with new tokens, all in parallel within an iteration.
• Training uses imitation learning with an expert policy (oracle or distilled teacher) to maximize log-likelihoods of deletion and insertion decisions (two objectives).
• Roll-in policies introduce noise or adversary outputs to diversify training states (mixture of ground-truth, model outputs, and expert corrections).
• Inference performs greedy decoding over multiple refinement iterations until convergence or timeout; termination includes loop detection and a penalty for empty placeholders.

Experimental results

Research questions

• RQ1 Can insertion and deletion operations be combined in a unified model to generate and refine sequences effectively?
• RQ2 How does dual-policy imitation learning perform for training an editing-based sequence generator?
• RQ3 Does LevT achieve comparable generation quality to autoregressive transformers while offering decoding efficiency gains?
• RQ4 Can a LevT trained on machine translation be directly applied to translation post-editing?

Key findings

• LevT achieves comparable or better BLEU/ROUGE scores than Transformer baselines on machine translation and text summarization.
• LevT delivers up to around 5x speed-up in actual decoding time due to parallelizable refinement steps.
• Weight sharing among the three heads generally helps, with insertion-related heads benefiting from shared representations.
• Dual-policy imitation learning with adversarial roll-ins improves training signals beyond a plain DAE-like setup.
• A LevT trained on MT can be applied to translation post-editing without changes (zero-shot) and further improves when fine-tuned on PE data.