[Paper Review] Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors

Wanyu Du, Jianqiao Zhao+2 more|arXiv (Cornell University)|Apr 4, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a variational encoder-decoder model with Gaussian process (GP) priors to enhance text generation diversity and quality by modeling dependent random context variables from deterministic encoder hidden states. By using a GP prior to induce structured dependencies among latent variables, the method enables diverse, context-aware generation without sacrificing quality, outperforming baselines in both paraphrase generation and style transfer tasks.

ABSTRACT

Generating high quality texts with high diversity is important for many NLG applications, but current methods mostly focus on building deterministic models to generate higher quality texts and do not provide many options for promoting diversity. In this work, we present a novel latent structured variable model to generate high quality texts by enriching contextual representation learning of encoder-decoder models. Specifically, we introduce a stochastic function to map deterministic encoder hidden states into random context variables. The proposed stochastic function is sampled from a Gaussian process prior to (1) provide infinite number of joint Gaussian distributions of random context variables (diversity-promoting) and (2) explicitly model dependency between context variables (accurate-encoding). To address the learning challenge of Gaussian processes, we propose an efficient variational inference approach to approximate the posterior distribution of random context variables. We evaluate our method in two typical text generation tasks: paraphrase generation and text style transfer. Experimental results on benchmark datasets demonstrate that our method improves the generation quality and diversity compared with other baselines.

Motivation & Objective

• To address the trade-off between text generation quality and diversity in existing autoencoder-based models.
• To improve contextual representation learning by modeling dependencies between latent variables in the encoder.
• To enable high-diversity text generation without relying solely on decoding strategies or fixed expert ensembles.
• To develop an efficient variational inference method for learning GP priors over variable-sized latent representations.
• To demonstrate the superiority of GP priors over independent priors and other baselines in quality and diversity metrics.

Proposed method

• Introduces a stochastic function mapping deterministic encoder hidden states to random context variables using a Gaussian process prior.
• Models the joint distribution of context variables via a GP prior to explicitly encode dependencies between them.
• Employs an amortized variational inference approach to approximate the posterior distribution of latent variables under the GP prior.
• Uses i.i.d. sampling from the variational posterior during inference to maintain computational efficiency.
• Applies the framework to both LSTM and Transformer-based encoder-decoder architectures for ablation and comparison.
• Leverages the GP prior to generate multiple diverse, high-quality outputs by sampling different context variable realizations.

Experimental results

Research questions

• RQ1 Can modeling dependencies between latent context variables improve text generation quality and diversity?
• RQ2 Does using a GP prior for latent variable modeling outperform independent priors in conditional text generation?
• RQ3 Can the proposed method alleviate posterior collapse while maintaining high generation quality?
• RQ4 How does the GP prior compare to decoding-only diversity strategies in terms of quality-diversity trade-offs?
• RQ5 To what extent does the method generalize across different architectures and text generation tasks?

Key findings

• GP-VAE achieves the highest generation quality on the GYAFC (E&M) test set among all baselines, outperforming PG + Normal prior and T-CVAE.
• For Transformer-based models, T5 + GP prior achieves the best performance in both token-level and sentence-level diversity metrics.
• The model significantly reduces posterior collapse compared to standard VAE baselines, especially in high-dimensional settings.
• On the style transfer task, the model generates diverse outputs while preserving semantic accuracy, as shown in qualitative examples.
• The method maintains the same inference-time complexity as standard VAEs due to i.i.d. sampling from the variational posterior.
• The GP prior enables context-aware variations that lead to more semantically coherent and diverse outputs than independent latent variable models.