[Paper Review] Monotonic Multihead Attention

Xutai Ma, Juan Pino+3 more|arXiv (Cornell University)|Sep 26, 2019

Topic Modeling17 references68 citations

TL;DR

The paper introduces Monotonic Multihead Attention (MMA), extending monotonic attention to Transformer-based models for simultaneous translation, with two variants (MMA-H and MMA-IL) and latency-control losses to improve quality-latency tradeoffs over MILk.

ABSTRACT

Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence. Recent approaches for this task either apply a fixed policy on a state-of-the art Transformer model, or a learnable monotonic attention on a weaker recurrent neural network-based structure. In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which extends the monotonic attention mechanism to multihead attention. We also introduce two novel and interpretable approaches for latency control that are specifically designed for multiple attentions heads. We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach. We also analyze how the latency controls affect the attention span and we motivate the introduction of our model by analyzing the effect of the number of decoder layers and heads on quality and latency.

Motivation & Objective

• Motivate online/real-time translation by enabling Transformer models to decode online without reading the full source.
• Develop a monotonic attention mechanism that supports multiple decoder heads and layers.
• Introduce latency control strategies to manage reading/writing schedules across heads and layers.
• Demonstrate superior latency-quality tradeoffs against MILk on standard benchmarks.
• Provide ablations to analyze the impact of decoder depth, heads, and the proposed loss terms on performance.

Proposed method

• Define MMA as per-head monotonic attention in each decoder layer, enabling parallel independent attention heads across layers (MMA-H and MMA-IL).
• Compute selection probabilities p_{i,j}^{l,h} via a monotonic energy with Sigmoid and Bernoulli sampling for each head in each layer.
• Use two variants: MMA-H (hard-attention with per-head hard alignment) and MMA-IL (infinite lookback with per-head softmax energy).
• Aggregate per-head contexts into a concatenated context for each layer and feed into the corresponding decoder block.
• Introduce latency regularizers: a differentiable average lagging-based loss L_avg and a head-divergence loss L_var to control reading speed and cross-head variance.
• Train with objective combining translation loss with λ_avg L_avg and λ_var L_var to balance quality and latency.

Experimental results

Research questions

• RQ1 Can monotonic attention be extended to multihead Transformer architectures for online decoding without sacrificing translation quality?
• RQ2 How do per-head latency controls influence attention spans and overall latency in MMA?
• RQ3 Do MMA-H and MMA-IL provide different quality-latency tradeoffs, and under what conditions?
• RQ4 What is the impact of the number of decoder layers and heads on quality and latency in MMA?
• RQ5 Do latency-controlling losses effectively prevent outlier heads from dominating reading speed?

Key findings

• MMA achieves better latency-quality tradeoffs than MILk on IWSLT15 En-Vi and WMT15 De-En benchmarks.
• MMA-H often outperforms MILk in the latency-quality Pareto frontier despite each head attending to a single state.
• The differentiable latency losses effectively reduce attention span and balance head speeds, as shown by shorter average attention spans with increased L_var.
• Increasing the number of decoder layers and heads generally improves BLEU but increases latency, indicating a tradeoff and need for tuning λ parameters.
• The proposed losses influence reading buffers and head diversity, reducing outlierHead-induced latency.