[Paper Review] Parallel Machine Translation with Disentangled Context Transformer

Jungo Kasai, James Cross+2 more|arXiv (Cornell University)|Jan 15, 2020

Natural Language Processing Techniques23 citations

TL;DR

This paper proposes the Disentangled Context (DisCo) Transformer, a non-autoregressive neural machine translation model that generates all output tokens in parallel by conditioning each on arbitrary subsets of reference tokens, using attention masking to enable context disentanglement. It achieves state-of-the-art translation quality with significantly reduced decoding time across 7 translation directions.

ABSTRACT

State-of-the-art neural machine translation models generate a translation from left to right and every step is conditioned on the previously generated tokens. The sequential nature of this generation process causes fundamental latency in inference since we cannot generate multiple tokens in each sentence in parallel. We propose an attention-masking based model, called Disentangled Context (DisCo) transformer, that simultaneously generates all tokens given different contexts. The DisCo transformer is trained to predict every output token given an arbitrary subset of the other reference tokens. We also develop the parallel easy-first inference algorithm, which iteratively refines every token in parallel and reduces the number of required iterations. Our extensive experiments on 7 translation directions with varying data sizes demonstrate that our model achieves competitive, if not better, performance compared to the state of the art in non-autoregressive machine translation while significantly reducing decoding time on average. Our code is available at this https URL.

Motivation & Objective

• Address the latency bottleneck in autoregressive neural machine translation caused by left-to-right token generation.
• Enable parallel generation of all output tokens without sacrificing translation quality.
• Develop a training paradigm that conditions each output token on diverse subsets of reference tokens to improve robustness and generalization.
• Design an efficient inference algorithm that iteratively refines all tokens in parallel to accelerate convergence.
• Demonstrate strong performance across diverse translation directions and data sizes, outperforming or matching state-of-the-art non-autoregressive models in both accuracy and speed.

Proposed method

• Propose an attention-masking mechanism that allows each output token to attend to any subset of other reference tokens, enabling disentangled context modeling.
• Train the model to predict every output token given an arbitrary subset of the reference sequence, promoting robustness to partial context.
• Design the parallel easy-first inference algorithm, which iteratively refines all tokens in parallel, reducing the number of required decoding iterations.
• Use a standard transformer architecture with modified cross-attention mechanisms to support the disentangled context setup.
• Apply a curriculum learning strategy during training, gradually increasing the complexity of the context subsets used for supervision.
• Leverage standard sequence-to-sequence training objectives with standard tokenization and normalization, while focusing on the attention masking and training objective design.

Experimental results

Research questions

• RQ1 Can a non-autoregressive transformer model achieve competitive translation quality while enabling full parallel generation of output tokens?
• RQ2 How does conditioning each output token on arbitrary subsets of reference tokens improve model robustness and performance?
• RQ3 To what extent does the proposed parallel easy-first inference algorithm reduce decoding time compared to standard non-autoregressive or autoregressive methods?
• RQ4 How does the model perform across diverse translation directions and varying data sizes compared to existing non-autoregressive baselines?
• RQ5 Does the disentangled context mechanism lead to better generalization and faster convergence during inference?

Key findings

• The DisCo Transformer achieves competitive or better translation performance than state-of-the-art non-autoregressive models across 7 translation directions.
• The model significantly reduces decoding time on average, demonstrating the effectiveness of the parallel easy-first inference strategy.
• The attention-masking based training scheme enables robust prediction of each output token using diverse context subsets, improving generalization.
• The model maintains strong performance even with limited training data, indicating strong inductive bias from the disentangled context design.
• The parallel inference algorithm reduces the number of required iterations by efficiently refining all tokens simultaneously.
• The approach generalizes well across diverse language pairs, showing consistent improvements in inference speed without major performance degradation.