[Paper Review] Sequence to Sequence Learning with Neural Networks

Ilya Sutskever, Oriol Vinyals+1 more|arXiv (Cornell University)|Sep 10, 2014

Natural Language Processing Techniques29 references13,314 citations

TL;DR

A neural sequence-to-sequence model using a dual LSTM encoder-decoder achieves state-of-the-art BLEU scores on WMT’14 English-French translation, outperforming a phrase-based SMT baseline in direct translation and improving rescoring when combined with SMT outputs.

ABSTRACT

Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT'14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous best result on this task. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.

Motivation & Objective

• Demonstrate an end-to-end sequence-to-sequence learning approach for mapping input sequences to output sequences without strong structural assumptions.
• Show that a deep LSTM encoder-decoder can translate text directly and improve SMT performance via rescoring.
• Investigate techniques to improve learning and translation quality, including reversing source sentences and using multi-layer architectures.

Proposed method

• Encode the input sequence with a deep LSTM to obtain a fixed-dimensional vector representation.
• Decode the target sequence with a separate deep LSTM conditioned on the encoded representation.
• Use a left-to-right beam-search decoder to generate translations and compute p(T|S) for translations.
• Train end-to-end by maximizing the log probability of correct translations across the training data.
• Experiment with reversing source sentences to reduce memory lag and improve optimization.
• Evaluate using BLEU on WMT’14 English-French, including direct translation and rescoring of SMT n-best lists.

Experimental results

Research questions

• RQ1 Can a fully neural encoder-decoder with LSTMs perform direct sequence-to-sequence translation at scale?
• RQ2 Does reversing the source input improve learning efficiency and translation quality in seq2seq LSTM models?
• RQ3 How does neural seq2seq translation compare to and complement traditional SMT baselines on a large-scale task?

Key findings

• An ensemble of deep LSTMs achieved 34.81 BLEU on ntst14 for direct translation, outperforming the SMT baseline’s 33.30 BLEU.
• Rescoring the SMT baseline’s 1000-best list with an ensemble of reversed LSTMs reached 36.5 BLEU, close to the best published SMT result.
• Single LSTM and various beam settings show neural models can approach or exceed SMT performance, especially when using reversal and ensembles.
• Reversing source sentences dramatically improves BLEU (from 25.9 to 30.6 in one setup) and perplexity (5.8 to 4.7).
• The complete model used 384M parameters with 160k source and 80k target vocabularies, trained with SGD and gradient clipping, across 7.5 epochs.
• Long sentences did not degrade performance; qualitative analyses show learned representations respect word order and capture meaning.