[Paper Review] Fast Interleaved Bidirectional Sequence Generation

Biao Zhang, Ivan Titov+1 more|arXiv (Cornell University)|Oct 27, 2020

Natural Language Processing Techniques55 references4 citations

TL;DR

This paper proposes the Interleaved Bidirectional Decoder (IBDecoder), a method that accelerates neural sequence generation by simultaneously generating tokens from both left-to-right and right-to-left directions using interleaved sequence reordering and modified self-attention masks. It achieves up to 11× speedup over autoregressive decoding with comparable or better translation and summarization quality, outperforming left-to-right semi-autoregressive decoding due to more favorable independence assumptions between distant, non-adjacent tokens.

ABSTRACT

Independence assumptions during sequence generation can speed up inference, but parallel generation of highly inter-dependent tokens comes at a cost in quality. Instead of assuming independence between neighbouring tokens (semi-autoregressive decoding, SA), we take inspiration from bidirectional sequence generation and introduce a decoder that generates target words from the left-to-right and right-to-left directions simultaneously. We show that we can easily convert a standard architecture for unidirectional decoding into a bidirectional decoder by simply interleaving the two directions and adapting the word positions and self-attention masks. Our interleaved bidirectional decoder (IBDecoder) retains the model simplicity and training efficiency of the standard Transformer, and on five machine translation tasks and two document summarization tasks, achieves a decoding speedup of ~2X compared to autoregressive decoding with comparable quality. Notably, it outperforms left-to-right SA because the independence assumptions in IBDecoder are more felicitous. To achieve even higher speedups, we explore hybrid models where we either simultaneously predict multiple neighbouring tokens per direction, or perform multi-directional decoding by partitioning the target sequence. These methods achieve speedups to 4X-11X across different tasks at the cost of <1 BLEU or <0.5 ROUGE (on average). Source code is released at https://github.com/bzhangGo/zero.

Motivation & Objective

• To address the slow inference speed of autoregressive sequence generation in neural models.
• To reduce the quality degradation caused by independence assumptions in semi-autoregressive decoding.
• To explore bidirectional generation as a more effective alternative to unidirectional or semi-autoregressive approaches.
• To enable efficient, high-speed decoding without adding model parameters or significant training overhead.
• To achieve higher speedups than prior methods while maintaining competitive generation quality.

Proposed method

• The IBDecoder reorganizes the target sequence by interleaving tokens from left-to-right and right-to-left generation directions into a single sequence.
• It modifies word positions and self-attention masks to allow intra- and inter-directional attention while preserving standard Transformer decoder compatibility.
• Independence is assumed between corresponding positions in the two directions (e.g., y₁ and yₙ), enabling parallel generation.
• The method is compatible with standard unidirectional decoders like the Transformer, requiring only sequence reordering and mask adjustments.
• Hybrid variants extend IBDecoder with multi-word prediction (SA) or multi-directional partitioning (IMDecoder) for further speedups.
• The approach is trained end-to-end with no additional parameters or complex data construction.

Experimental results

Research questions

• RQ1 Can bidirectional sequence generation be efficiently integrated into standard Transformer decoders without architectural changes?
• RQ2 Does interleaving left-to-right and right-to-left generation reduce quality degradation compared to left-to-right semi-autoregressive methods?
• RQ3 Can the independence assumptions in bidirectional generation be less harmful than those in unidirectional semi-autoregressive decoding?
• RQ4 What speedups can be achieved with minimal quality loss using this approach across diverse sequence generation tasks?
• RQ5 How does the method scale with multi-word prediction or multi-directional partitioning strategies?

Key findings

• The IBDecoder achieves a 2× speedup over autoregressive decoding on five machine translation and two summarization tasks with comparable BLEU and ROUGE scores.
• On average, IBDecoder with semi-autoregressive extension (IBDecoder+SA) achieves 4.2× to 11.15× speedup across tasks, with less than 1 BLEU or 0.5 ROUGE point drop.
• The model outperforms left-to-right semi-autoregressive decoding in quality, demonstrating that bidirectional independence assumptions are less harmful than unidirectional ones.
• With knowledge distillation, IBDecoder surpasses the standard Transformer baseline on five out of six tasks in terms of BLEU and ROUGE scores.
• The method achieves 3.02× speedup on the CDMail summarization task, a long-sequence benchmark where non-autoregressive models typically underperform.
• The approach is orthogonal to other speedup techniques, such as model compression, and can be combined with them for further gains.