[Paper Review] Paraphrase Generation with Latent Bag of Words

Yao Fu, Yansong Feng+1 more|arXiv (Cornell University)|Jan 7, 2020

Topic Modeling35 citations

TL;DR

The paper introduces a latent bag-of-words (BOW) model grounded to target sentences to guide differentiable paraphrase generation, enabling interpretable content planning and improved generation performance.

ABSTRACT

Paraphrase generation is a longstanding important problem in natural language processing. In addition, recent progress in deep generative models has shown promising results on discrete latent variables for text generation. Inspired by variational autoencoders with discrete latent structures, in this work, we propose a latent bag of words (BOW) model for paraphrase generation. We ground the semantics of a discrete latent variable by the BOW from the target sentences. We use this latent variable to build a fully differentiable content planning and surface realization model. Specifically, we use source words to predict their neighbors and model the target BOW with a mixture of softmax. We use Gumbel top-k reparameterization to perform differentiable subset sampling from the predicted BOW distribution. We retrieve the sampled word embeddings and use them to augment the decoder and guide its generation search space. Our latent BOW model not only enhances the decoder, but also exhibits clear interpretability. We show the model interpretability with regard to \emph{(i)} unsupervised learning of word neighbors \emph{(ii)} the step-by-step generation procedure. Extensive experiments demonstrate the transparent and effective generation process of this model.\footnote{Our code can be found at \url{https://github.com/FranxYao/dgm_latent_bow}}

Motivation & Objective

• Ground the latent variable to explicit lexical semantics using the target sentence BOW.
• Build a differentiable content planning and surface realization pipeline for paraphrase generation.
• Learn word neighbors unsupervised and demonstrate interpretability through step-by-step generation.

Proposed method

• Model a hierarchical discrete latent variable z as a bag of words from the target sentences.
• Predict neighbor words for each source word and form a mixed neighbor BOW via a mixture of softmax.
• Sample a k-hot BOW z from p(tilde z|x) using differentiable Gumbel top-k reparameterization.
• Augment the decoder with sampled word embeddings to steer generation.
• Train with a joint loss combining sequence-to-sequence likelihood and BOW guidance.
• Leverage a mixture of softmax to capture multimodal BOW distributions.

Experimental results

Research questions

• RQ1 Can grounding a latent discrete variable to the target BOW improve paraphrase generation while remaining end-to-end differentiable?
• RQ2 Does differentiable subset sampling via Gumbel top-k enable effective planning and realization in paraphrase generation?
• RQ3 To what extent does latent BOW prediction influence decoding and generation quality?

Key findings

• LBOW models (Topk and Gumbel) achieve higher BLEU/ROUGE than standard seq2seq baselines on Quora and MSCOCO.
• Gumbel-top-k provides differentiable sampling of the BOW, enabling end-to-end training.
• BOW-guided decoding significantly constrains the search space and improves generation quality.
• The model learns word neighbors and lexical SEMANTIC relations in an unsupervised manner.
• A cheating BOW upper bound confirms the importance of accurate BOW prediction for strong performance.