[Paper Review] Towards Universal Paraphrastic Sentence Embeddings

John Wieting, Mohit Bansal+2 more|arXiv (Cornell University)|Nov 25, 2015

Topic Modeling64 references117 citations

TL;DR

This paper proposes a simple yet highly effective method for learning universal sentence embeddings by averaging word vectors trained on the Paraphrase Database (PPDB), achieving state-of-the-art performance on cross-domain textual similarity and entailment tasks. Despite its simplicity, the model outperforms complex LSTMs on out-of-domain data and rivals or exceeds task-specific systems, establishing a new baseline for universal sentence embeddings without requiring neural network architectures.

ABSTRACT

We consider the problem of learning general-purpose, paraphrastic sentence embeddings based on supervision from the Paraphrase Database (Ganitkevitch et al., 2013). We compare six compositional architectures, evaluating them on annotated textual similarity datasets drawn both from the same distribution as the training data and from a wide range of other domains. We find that the most complex architectures, such as long short-term memory (LSTM) recurrent neural networks, perform best on the in-domain data. However, in out-of-domain scenarios, simple architectures such as word averaging vastly outperform LSTMs. Our simplest averaging model is even competitive with systems tuned for the particular tasks while also being extremely efficient and easy to use. In order to better understand how these architectures compare, we conduct further experiments on three supervised NLP tasks: sentence similarity, entailment, and sentiment classification. We again find that the word averaging models perform well for sentence similarity and entailment, outperforming LSTMs. However, on sentiment classification, we find that the LSTM performs very strongly-even recording new state-of-the-art performance on the Stanford Sentiment Treebank. We then demonstrate how to combine our pretrained sentence embeddings with these supervised tasks, using them both as a prior and as a black box feature extractor. This leads to performance rivaling the state of the art on the SICK similarity and entailment tasks. We release all of our resources to the research community with the hope that they can serve as the new baseline for further work on universal sentence embeddings.

Motivation & Objective

• To develop general-purpose, paraphrastic sentence embeddings that transfer effectively across diverse NLP domains.
• To evaluate the performance of various compositional architectures—ranging from simple averaging to LSTMs—on in-domain and out-of-domain textual similarity tasks.
• To determine whether simple, non-neural models can outperform complex neural architectures in zero-shot or few-shot transfer settings.
• To demonstrate that pretrained sentence embeddings can improve performance on downstream NLP tasks like similarity, entailment, and sentiment classification.
• To release a new, accessible baseline for universal sentence embeddings to accelerate future research.

Proposed method

• Training sentence embeddings by averaging word vectors learned on the Paraphrase Database (PPDB), with no additional compositional parameters beyond the word vectors themselves.
• Using a modified version of paragram-sl999 word embeddings, fine-tuned via backpropagation on PPDB phrase pairs to produce paragram-phrase embeddings.
• Applying a learned multiplicative weight to each word vector based on the L2 norm of its corresponding paragram-phrase embedding to emphasize important content words.
• Evaluating models on in-domain (SICK) and out-of-domain (22 SemEval STS) datasets to assess transferability and robustness.
• Combining pretrained sentence embeddings as a prior or fixed feature extractor in supervised models for similarity, entailment, and sentiment classification.
• Using cosine similarity between sentence embeddings as the primary metric for evaluating paraphrase detection and textual similarity.

Experimental results

Research questions

• RQ1 Can a simple word averaging model outperform complex neural architectures like LSTMs in cross-domain sentence similarity transfer?
• RQ2 How does the performance of sentence embeddings trained on paraphrase data generalize across diverse domains such as news, tweets, and image captions?
• RQ3 To what extent does learning word vector compositionality within the averaging framework improve performance over simply averaging pretrained word embeddings?
• RQ4 Can universal sentence embeddings serve as an effective prior or feature extractor in supervised NLP tasks like entailment and sentiment classification?
• RQ5 What role do word importance weights derived from embedding norms play in enhancing sentence representation quality?

Key findings

• The word averaging model, despite its simplicity, achieved an average Pearson’s $r$ of 66.83 across 22 SemEval STS datasets, outperforming LSTMs by 16.5 points on average.
• The paragram-phrase embeddings placed in the top 25% of all systems in every SemEval STS task from 2012 to 2015, achieving the best or tied-for-best performance on four datasets.
• The model outperformed both GloVe and paragram-sl999 embeddings when averaged, with a 17.1-point improvement over GloVe and 12.8-point improvement over paragram-sl999 on average.
• The learned multiplicative weights, derived from the L2 norms of paragram-phrase vectors, accounted for at least 64.76% of the performance gain over raw paragram-sl999 embeddings.
• On the Stanford Sentiment Treebank, the LSTM model achieved a new state-of-the-art accuracy of 89.2% on the coarse-grained sentiment classification task, outperforming the averaging model in this specific task.
• When used as a fixed feature extractor or prior, the pretrained sentence embeddings achieved performance competitive with state-of-the-art models on the SICK similarity and entailment tasks.