[Paper Review] A Deep Generative Framework for Paraphrase Generation

Ankush Gupta, Arvind Agarwal+2 more|arXiv (Cornell University)|Sep 15, 2017

Topic Modeling134 citations

TL;DR

The paper presents a VAE-LSTM based framework that conditions both encoder and decoder on the original sentence to generate multiple paraphrases for a given input, achieving strong improvements over baselines on MSCOCO and Quora datasets. It also provides human evaluations showing well-formed and relevant paraphrases.

ABSTRACT

Paraphrase generation is an important problem in NLP, especially in question answering, information retrieval, information extraction, conversation systems, to name a few. In this paper, we address the problem of generating paraphrases automatically. Our proposed method is based on a combination of deep generative models (VAE) with sequence-to-sequence models (LSTM) to generate paraphrases, given an input sentence. Traditional VAEs when combined with recurrent neural networks can generate free text but they are not suitable for paraphrase generation for a given sentence. We address this problem by conditioning the both, encoder and decoder sides of VAE, on the original sentence, so that it can generate the given sentence's paraphrases. Unlike most existing models, our model is simple, modular and can generate multiple paraphrases, for a given sentence. Quantitative evaluation of the proposed method on a benchmark paraphrase dataset demonstrates its efficacy, and its performance improvement over the state-of-the-art methods by a significant margin, whereas qualitative human evaluation indicate that the generated paraphrases are well-formed, grammatically correct, and are relevant to the input sentence. Furthermore, we evaluate our method on a newly released question paraphrase dataset, and establish a new baseline for future research.

Motivation & Objective

• Motivate and address paraphrase generation for NLP applications such as QA, IR, and information extraction.
• Develop a principled deep generative framework that can produce multiple high-quality paraphrases for a given sentence.
• Leverage and extend VAE with conditioning to reflect the input sentence in both encoder and decoder sides.
• Demonstrate improvements over state-of-the-art on standard paraphrase datasets and establish new baselines on a question paraphrase dataset.

Proposed method

• Use a variational autoencoder (VAE) where both the encoder and decoder are conditioned on the original input sentence.
• Represent input and paraphrase sentences with LSTM encoders, and generate paraphrase representations via a latent variable z.
• In the decoder, initialize with the original sentence representation and inject z at each decoding stage to reconstruct the paraphrase representation.
• Train with a variational lower bound that conditions on the input sentence: L = E_{q(z|xo, xp)}[log p(xp|z, xo)] - KL(q(z|xo, xp) || p(z)).
• Compare supervised variants (VAE-SVG and VAE-SVG-eq) against unsupervised VAE baselines and state-of-the-art models on MSCOCO and Quora datasets.

Experimental results

Research questions

• RQ1 Can a VAE-based architecture be conditioned on the input sentence to generate paraphrases specifically for that sentence?
• RQ2 Do conditioning and the proposed VAE-LSTM design improve paraphrase quality over unconditional VAEs and standard seq-to-seq models?
• RQ3 How do the proposed models perform on general paraphrase data (MSCOCO) and question paraphrase data (Quora) in terms of automatic metrics and human judgments?

Key findings

• The VAE-SVG family outperforms baselines and state-of-the-art methods on MSCOCO in BLEU, METEOR, and TER metrics under various settings.
• On the Quora dataset, supervised variants (VAE-SVG and VAE-SVG-eq) significantly beat unsupervised baselines, with further gains from beam-search in certain configurations.
• Human evaluation shows paraphrases are largely relevant and readable, approaching ground-truth quality, with some domain-dependent variance (MSCOCO captions).
• The approach establishes strong baselines on the 50K/100K/150K training regimes for Quora and demonstrates substantial improvements over prior methods for question paraphrase generation.
• Beam-search and variant encodings (notably VAE-SVG-eq) contribute to higher quality paraphrases and more effective filtering based on input-variant similarity.