[Paper Review] Learned in Translation: Contextualized Word Vectors

Bryan McCann, James Bradbury+2 more|arXiv (Cornell University)|Aug 1, 2017

Topic Modeling240 citations

TL;DR

This paper introduces CoVe, context vectors derived from an MT-trained MT-LSTM encoder, and shows that appending CoVe to word embeddings improves performance across multiple NLP tasks (sentiment, entailment, QA) over word vectors alone.

ABSTRACT

Computer vision has benefited from initializing multiple deep layers with weights pretrained on large supervised training sets like ImageNet. Natural language processing (NLP) typically sees initialization of only the lowest layer of deep models with pretrained word vectors. In this paper, we use a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation (MT) to contextualize word vectors. We show that adding these context vectors (CoVe) improves performance over using only unsupervised word and character vectors on a wide variety of common NLP tasks: sentiment analysis (SST, IMDb), question classification (TREC), entailment (SNLI), and question answering (SQuAD). For fine-grained sentiment analysis and entailment, CoVe improves performance of our baseline models to the state of the art.

Motivation & Objective

• Motivate transfer learning in NLP by leveraging encoder representations trained on machine translation, analogous to pretrained CNNs in vision.
• Develop a reusable MT-based encoder that can provide token-level context vectors for downstream tasks.
• Demonstrate that augmenting word embeddings with CoVe improves performance on classification, entailment, and question answering tasks.
• Investigate how MT data quantity and quality affect downstream gains from CoVe.

Proposed method

• Train a two-layer bidirectional LSTM encoder (MT-LSTM) on English→German MT data to produce context representations for each input token.
• Define CoVe(w) as MT-LSTM(GloVe(w)) and concatenate CoVe with GloVe word vectors for downstream models.
• Develop a general biattentive classification network (BCN) that uses biattention and pooling to produce task representations from input sequences enhanced with CoVe.
• Adapt the Dynamic Coattention Network framework for question answering to incorporate CoVe.
• Experiment with MT-LSTMs trained on MT-Small, MT-Medium, and MT-Large to study data-size effects on transfer.
• Compare CoVe against baseline word vectors and character n-gram embeddings, and test on SST, SNLI, SQuAD, TREC, and IMDb.

Experimental results

Research questions

• RQ1 Does incorporating CoVe improve downstream NLP task performance compared to baseline pretrained word vectors alone?
• RQ2 How does the size/quality of MT training data influence the effectiveness of CoVe on various tasks?
• RQ3 Are CoVe representations complementary to word vectors and character-level features across different NLP tasks?
• RQ4 Can the MT-trained encoder be effectively transferred across single- and multi-sentence classification, entailment, and QA tasks?

Key findings

• CoVe consistently improves validation performance when appended to GloVe vectors across multiple tasks.
• CoVe provides larger gains than using character n-gram embeddings alone, and combining CoVe with character features yields further improvements on some tasks.
• MT-LSTMs trained on larger MT datasets (MT-Large) give stronger downstream gains than smaller datasets.
• State-of-the-art results are achieved on SST-2, SST-5, and SNLI with CoVe in the reported configurations.
• In SQuAD, adding CoVe to a strong model (DCN) improves exact match and F1 relative to several baselines.