[논문 리뷰] Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, Bart van Merriënboer외 5명|arXiv (Cornell University)|2014. 06. 03.

Natural Language Processing Techniques참고 문헌 27인용 수 3,319

한 줄 요약

요약: RNN 인코더–디코더를 도입하여 SMT에서 가변 길이 시퀀스를 고정 길이 표현으로 매핑하고, 신경 언어 모델과 결합 시 BLEU를 향상시킵니다.

ABSTRACT

In this paper, we propose a novel neural network model called RNN Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder-Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases.

연구 동기 및 목표

• 뉴럴 시퀀스-투-시퀀스 모델을 사용해 SMT를 위한 문구 표현을 학습하는 동기를 부여한다.
• 소스 문구에 조건화된 타깃 문구를 점수화하기 위해 공동으로 학습된 RNN 인코더–디코더를 제안한다.
• 로그-선형 모델에 추가 특성으로써 RNN 기반 점수가 SMT 성능을 향상시키는지 보여준다.
• 학습된 문구 표현이 의미적·통사적 구조를 포착한다는 것을 입증한다.]
• method_1: [
• Propose an RNN Encoder–Decoder with an encoder RNN mapping source sequences to a fixed-length vector c and a decoder RNN generating target sequences conditioned on c and previous outputs.
• Introduce a novel hidden unit with reset and update gates to adaptively remember and forget information (an LSTM-inspired, simplified variant).
• Train the model to maximize the conditional log-likelihood log p(y | x) jointly over (x, y) pairs.
• Use the trained encoder–decoder to score phrase pairs in a phrase table and incorporate these scores as extra features in a log-linear SMT framework.
• Compare with a neural language model-based approach (CSLM) and with baseline phrase-based SMT using BLEU as the evaluation metric.

제안 방법

• Propose an RNN Encoder–Decoder with an encoder RNN mapping source sequences to a fixed-length vector c and a decoder RNN generating target sequences conditioned on c and previous outputs.
• Introduce a novel hidden unit with reset and update gates to adaptively remember and forget information (an LSTM-inspired, simplified variant).
• Train the model to maximize the conditional log-likelihood log p(y | x) jointly over (x, y) pairs.
• Use the trained encoder–decoder to score phrase pairs in a phrase table and incorporate these scores as extra features in a log-linear SMT framework.
• Compare with a neural language model-based approach (CSLM) and with baseline phrase-based SMT using BLEU as the evaluation metric.

실험 결과

연구 질문

• RQ1 Does an RNN Encoder–Decoder trained on phrase pairs provide useful scores for SMT beyond traditional translation probabilities?
• RQ2 Do neural scores from the RNN Encoder–Decoder improve BLEU when integrated into a standard SMT pipeline?
• RQ3 What kind of linguistic regularities and representations does the RNN Encoder–Decoder learn for phrases?
• RQ4 Are the learned phrase representations complementary to neural language models in SMT performance?
• RQ5 Can the model reveal semantic and syntactic structure in learned phrase embeddings?

주요 결과

모델	BLEU dev	BLEU test
Baseline	30.64	33.30
RNN	31.20	33.87
CSLM + RNN	31.48	34.64
CSLM + RNN + WP	31.50	34.54

• Adding RNN Encoder–Decoder scores to the baseline SMT system improves BLEU on development and test sets.
• The best BLEU results occur when combining CSLM (neural language model) with RNN Encoder–Decoder scores.
• Penalizing unknown words in the neural features did not improve test BLEU, but affected development BLEU.
• Qualitative analysis shows the RNN Encoder–Decoder captures linguistic regularities and tends to propose well-formed target phrases.
• Word and phrase representations learned by the model form meaningful semantic clusters consistent with linguistic structure.