[Paper Review] Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations

John Wieting, Kevin Gimpel|arXiv (Cornell University)|Nov 15, 2017

Topic Modeling38 references23 citations

TL;DR

This paper introduces ParaNMT-50M, a dataset of 51 million English-english paraphrase pairs generated via back-translation of a large parallel corpus. Using this corpus, the authors train paraphrastic sentence embeddings that outperform all prior supervised systems on SemEval STS benchmarks and achieve state-of-the-art performance in capturing semantic similarity across diverse text styles and structures.

ABSTRACT

We extend the work of Wieting et al. (2017), back-translating a large parallel corpus to produce a dataset of more than 51 million English-English sentential paraphrase pairs in a dataset we call ParaNMT-50M. We find this corpus to be cover many domains and styles of text, in addition to being rich in paraphrases with different sentence structure, and we release it to the community. To show its utility, we use it to train paraphrastic sentence embeddings using only minor changes to the framework of Wieting et al. (2016b). The resulting embeddings outperform all supervised systems on every SemEval semantic textual similarity (STS) competition, and are a significant improvement in capturing paraphrastic similarity over all other sentence embeddings We also show that our embeddings perform competitively on general sentence embedding tasks. We release the corpus, pretrained sentence embeddings, and code to generate them. We believe the corpus can be a valuable resource for automatic paraphrase generation and can provide a rich source of semantic information to improve downstream natural language understanding tasks.

Motivation & Objective

• To create a large-scale, diverse, and high-quality paraphrase dataset covering multiple domains and text styles.
• To improve the performance of sentence embeddings in capturing paraphrastic similarity beyond existing supervised methods.
• To demonstrate the utility of back-translation for generating large-scale paraphrase data without human annotation.
• To release a resource that supports automatic paraphrase generation and enhances downstream NLU tasks.

Proposed method

• Back-translation of a large parallel monolingual corpus to generate synthetic English-English paraphrase pairs.
• Leveraging the framework of Wieting et al. (2016b) with minor modifications to train paraphrastic sentence embeddings.
• Using the resulting 51 million paraphrase pairs to pre-train sentence embeddings on a large-scale, diverse distribution of text.
• Applying the same training procedure to achieve strong performance on both paraphrase and general sentence embedding tasks.
• Releasing the full ParaNMT-50M dataset, pretrained embeddings, and code for reproducibility and community use.

Experimental results

Research questions

• RQ1 Can back-translation of a large parallel corpus generate a high-quality, large-scale paraphrase dataset suitable for training robust sentence embeddings?
• RQ2 How does the performance of sentence embeddings trained on ParaNMT-50M compare to existing supervised systems on semantic textual similarity tasks?
• RQ3 To what extent does the diversity of text styles and domains in ParaNMT-50M improve the generalization of sentence embeddings?
• RQ4 Can the proposed embeddings achieve competitive performance on general sentence embedding benchmarks beyond paraphrase tasks?

Key findings

• The ParaNMT-50M dataset contains over 51 million English-English paraphrase pairs, covering diverse domains and text styles.
• Sentence embeddings trained on ParaNMT-50M outperform all prior supervised systems on every SemEval STS competition task.
• The embeddings achieve a significant improvement in capturing paraphrastic similarity compared to all other sentence embedding models.
• The embeddings perform competitively on general sentence embedding tasks, indicating broad applicability.
• The release of the dataset, pretrained embeddings, and code enables widespread use in paraphrase generation and downstream NLU applications.