[Paper Review] Convolutional Sequence to Sequence Learning

Jonas Gehring, Michael Auli+3 more|arXiv (Cornell University)|May 8, 2017

Topic Modeling37 references1,901 citations

TL;DR

Introduces a fully convolutional sequence-to-sequence model with gated linear units and per-layer attention, achieving state-of-the-art BLEU on multiple translation tasks and faster training/inference than RNN-based rivals.

ABSTRACT

The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT'14 English-French translation at an order of magnitude faster speed, both on GPU and CPU.

Motivation & Objective

• Demonstrate that a fully convolutional encoder-decoder can outperform recurrent seq-to-seq models on large-scale translation tasks.
• Show that parallelizable convolutional architectures with gating and residual connections can model long-range dependencies effectively.
• Evaluate the impact of position embeddings, multi-step attention, and kernel depth/width on translation and summarization performance.
• Assess generation speed and scalability relative to strong RNN-based baselines.

Proposed method

• Use an entirely convolutional encoder-decoder architecture.
• Apply gated linear units (GLU) as non-linearities with residual connections.
• Incorporate multi-step attention in every decoder layer with input embeddings added to encoder outputs.
• Employ position embeddings to encode absolute input/output positions and apply padding strategies to maintain causality in the decoder.
• Train with a token-level likelihood objective and use beam search during inference.
• Evaluate on WMT’16 English-Romanian, WMT’14 English-German, WMT’14 English-French, and abstractive summarization on Gigaword.

Experimental results

Research questions

• RQ1 Can a fully convolutional seq-to-seq model match or surpass state-of-the-art RNN-based models on large-scale translation benchmarks?
• RQ2 Does incorporating multi-layer attention in every decoder layer improve translation quality and training stability?
• RQ3 What is the impact of position embeddings, kernel width/depth, and training strategies on performance and speed?
• RQ4 How does the ConvS2S model perform in terms of generation speed on GPU/CPU versus GNMT-like architectures?
• RQ5 Can the approach extend effectively to abstractive summarization tasks?

Key findings

• ConvS2S outperforms the deep LSTM setup of Wu et al. (2016) on WMT’14 English-German by 0.5 BLEU and on WMT’14 English-French by 1.6 BLEU in comparable settings.
• On WMT’16 English-Romanian with BPE vocabularies, ConvS2S achieves 29.45–30.02 BLEU compared to 28.1 BLEU for the prior winner.
• ConvS2S achieves new state-of-the-art on WMT’16 English-Romanian, WMT’14 English-German, and WMT’14 English-French translation tasks, and translates unseen sentences at an order of magnitude faster speed than Wu et al. (2016) across GPU/CPU.
• Multi-layer attention (all decoder layers) yields the best perplexity and BLEU, with small overhead for attention modules.
• Increasing encoder depth yields progressive accuracy gains, while deeper decoders provide diminishing returns; narrow kernels with more layers often outperform wider kernels.
• Ensemble of eight ConvS2S models outperforms previous strong ensembles on WMT’14 English-German and English-French.