[Paper Review] Neural Machine Translation in Linear Time

Nal Kalchbrenner, Lasse Espeholt+4 more|arXiv (Cornell University)|Oct 31, 2016

Natural Language Processing Techniques10 references317 citations

TL;DR

The ByteNet introduces a 1D dilated CNN encoder-decoder for neural translation with linear time overall, preserving source resolution and enabling dynamic unfolding for variable-length sequences, achieving state-of-the-art results on character-level modelling and English-German character-based translation.

ABSTRACT

We present a novel neural network for processing sequences. The ByteNet is a one-dimensional convolutional neural network that is composed of two parts, one to encode the source sequence and the other to decode the target sequence. The two network parts are connected by stacking the decoder on top of the encoder and preserving the temporal resolution of the sequences. To address the differing lengths of the source and the target, we introduce an efficient mechanism by which the decoder is dynamically unfolded over the representation of the encoder. The ByteNet uses dilation in the convolutional layers to increase its receptive field. The resulting network has two core properties: it runs in time that is linear in the length of the sequences and it sidesteps the need for excessive memorization. The ByteNet decoder attains state-of-the-art performance on character-level language modelling and outperforms the previous best results obtained with recurrent networks. The ByteNet also achieves state-of-the-art performance on character-to-character machine translation on the English-to-German WMT translation task, surpassing comparable neural translation models that are based on recurrent networks with attentional pooling and run in quadratic time. We find that the latent alignment structure contained in the representations reflects the expected alignment between the tokens.

Motivation & Objective

• Motivate neural translation models that scale linearly with sequence length and avoid heavy memorization.
• Develop a resolution-preserving encoder-decoder architecture that preserves temporal structure.
• Introduce dynamic unfolding to connect encoder representations to variable-length decoders.
• Use dilation in convolutions to expand receptive fields efficiently.
• Evaluate ByteNet on character-level language modelling and character-to-character translation tasks.

Proposed method

• Stack a CNN-based encoder and a CNN-based decoder on top of encoder representations to preserve resolution.
• Apply dynamic unfolding to generate variable-length outputs from a fixed encoder representation bound.
• Mask decoder convolutions to prevent access to future target tokens (causal masking).
• Use dilated convolutions with doubling dilation rates to enlarge receptive fields efficiently.
• Wrap layers in residual blocks with either ReLU-based or multiplicative units, with layer normalization.
• Train with Adam optimization and evaluate using bits-per-character for language modelling and BLEU for translation.

Experimental results

Research questions

• RQ1 Can a fully convolutional encoder-decoder with dynamic unfolding achieve competitive translation quality while maintaining linear-time complexity?
• RQ2 Does preserving the source sequence resolution and using dilated convolutions enable effective modelling of long-range dependencies in translation?
• RQ3 How does ByteNet perform on character-level language modelling compared to recurrent models?
• RQ4 What are the translation performance gains of ByteNet on English-to-German character-level/character-based translation benchmarks?
• RQ5 What does the learned latent alignment structure reveal about token-level correspondences in translations?

Key findings

• ByteNet achieves state-of-the-art on character-level language modelling (1.31 bits/character on Wikipedia).
• ByteNet attains state-of-the-art BLEU scores for character-to-character English-German translation on WMT NewsTest 2014 and 2015 (BLEU 23.75 and 26.26).
• On WMT 2015, ByteNet achieves the best published results to date among reported systems for character-level translation.
• ByteNet demonstrates linear running time with respect to input and output lengths and maintains a resolution-preserving source representation.
• The model shows latent alignment in representations that mirrors expected token alignments.
• Compared to recurrent models, ByteNet outperforms or matches strong baselines while avoiding quadratic-time computation.