[Paper Review] Topic-Guided Variational Autoencoders for Text Generation

Wenlin Wang, Zhe Gan+6 more|arXiv (Cornell University)|Mar 17, 2019

Topic ModelingComputer Science59 references57 citations

TL;DR

Introduces Topic-Guided Variational Autoencoder (TGVAE) with a topic-conditioned Gaussian mixture prior and Householder flows to improve both unconditional and conditional text generation. Demonstrates superior perplexity/BLEU and topic coherence across datasets.

ABSTRACT

We propose a topic-guided variational autoencoder (TGVAE) model for text generation. Distinct from existing variational autoencoder (VAE) based approaches, which assume a simple Gaussian prior for the latent code, our model specifies the prior as a Gaussian mixture model (GMM) parametrized by a neural topic module. Each mixture component corresponds to a latent topic, which provides guidance to generate sentences under the topic. The neural topic module and the VAE-based neural sequence module in our model are learned jointly. In particular, a sequence of invertible Householder transformations is applied to endow the approximate posterior of the latent code with high flexibility during model inference. Experimental results show that our TGVAE outperforms alternative approaches on both unconditional and conditional text generation, which can generate semantically-meaningful sentences with various topics.

Motivation & Objective

• Motivate topic-aware latent representations to guide text generation beyond simple Gaussian priors.
• Integrate a neural topic model to parameterize a Gaussian mixture prior over latent codes.
• Mitigate posterior collapse and increase posterior flexibility via Householder flows.
• Demonstrate improvements in both unconditional generation and conditional tasks like abstractive summarization.

Proposed method

• Specify a Gaussian mixture prior for latent z, where each component corresponds to a learned topic via a neural topic model (NTM).
• Use a neural sequence model (NSM) to decode sentences conditioned on z, with a GRU-based decoder.
• Employ Householder flows to construct a flexible approximate posterior q(z|y) by composing invertible, volume-preserving transformations.
• Derive a tractable ELBO combining topic-model terms and sequence-model terms, with an upper bound for KL between GMMs to enable training.
• Extend the model to text summarization by conditioning the decoder on source text x and incorporating topic guidance.
• Include a topic diversity regularizer to encourage distinct topics across the learned topics.

Experimental results

Research questions

• RQ1 Can a topic-guided Gaussian mixture prior improve latent space structure and sentence quality?
• RQ2 Does integrating a neural topic model with a VAE reduce posterior collapse and improve generation diversity and coherence?
• RQ3 How does Householder flow affect posterior flexibility and generation performance in TGVAE?
• RQ4 Can TGVAE improve both unconditional text generation and conditional tasks such as abstractive summarization?

Key findings

• TGVAE achieves lower perplexity upper bounds than several baselines on APNEWS, IMDB, and BNC datasets.
• Increasing the number of topics generally improves BLEU-based and self-BLEU diversity metrics for generated text.
• Householder flows (HF) enhance posterior inference and, with appropriate flow depth K, improve generation quality over standard VAE baselines.
• Topic-guided generation with a GMM prior yields better topic coherence scores (NPMI) across APNEWS, IMDB, and BNC.
• For summarization, injecting topic guidance into the Seq2Seq framework improves generation by leveraging semantic topic structure.