[Paper Review] Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, Bart van Merriënboer+5 more|arXiv (Cornell University)|Jun 3, 2014

Natural Language Processing Techniques27 references3,319 citations

TL;DR

Introduces an RNN Encoder–Decoder that maps variable-length sequences to fixed-length representations for scoring phrase pairs in SMT, improving BLEU when combined with neural language models.

ABSTRACT

In this paper, we propose a novel neural network model called RNN Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder-Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases.

Motivation & Objective

• Motivate the use of neural sequence-to-sequence models to learn phrase representations for SMT.
• Propose a jointly trained RNN Encoder–Decoder to score target phrases conditioned on source phrases.
• Show that RNN-based scores improve SMT performance when added as features in a log-linear model.
• Demonstrate that learned phrase representations capture semantic and syntactic structure.

Proposed method

• Propose an RNN Encoder–Decoder with an encoder RNN mapping source sequences to a fixed-length vector c and a decoder RNN generating target sequences conditioned on c and previous outputs.
• Introduce a novel hidden unit with reset and update gates to adaptively remember and forget information (an LSTM-inspired, simplified variant).
• Train the model to maximize the conditional log-likelihood log p(y | x) jointly over (x, y) pairs.
• Use the trained encoder–decoder to score phrase pairs in a phrase table and incorporate these scores as extra features in a log-linear SMT framework.
• Compare with a neural language model-based approach (CSLM) and with baseline phrase-based SMT using BLEU as the evaluation metric.

Experimental results

Research questions

• RQ1 Does an RNN Encoder–Decoder trained on phrase pairs provide useful scores for SMT beyond traditional translation probabilities?
• RQ2 Do neural scores from the RNN Encoder–Decoder improve BLEU when integrated into a standard SMT pipeline?
• RQ3 What kind of linguistic regularities and representations does the RNN Encoder–Decoder learn for phrases?
• RQ4 Are the learned phrase representations complementary to neural language models in SMT performance?
• RQ5 Can the model reveal semantic and syntactic structure in learned phrase embeddings?

Key findings

• Adding RNN Encoder–Decoder scores to the baseline SMT system improves BLEU on development and test sets.
• The best BLEU results occur when combining CSLM (neural language model) with RNN Encoder–Decoder scores.
• Penalizing unknown words in the neural features did not improve test BLEU, but affected development BLEU.
• Qualitative analysis shows the RNN Encoder–Decoder captures linguistic regularities and tends to propose well-formed target phrases.
• Word and phrase representations learned by the model form meaningful semantic clusters consistent with linguistic structure.