[Paper Review] Variational Transformers for Diverse Response Generation

Zhaojiang Lin, Genta Indra Winata+3 more|arXiv (Cornell University)|Mar 28, 2020

Speech Recognition and SynthesisComputer Science27 references46 citations

TL;DR

This paper introduces Variational Transformer (VT) models—Global Variational Transformer (GVT) and Sequential Variational Transformer (SVT)—to couple Transformer efficiency with CVAE-style latent variables for diverse, coherent dialogue responses, showing improvements over baselines in both automatic metrics and human judgments.

ABSTRACT

Despite the great promise of Transformers in many sequence modeling tasks (e.g., machine translation), their deterministic nature hinders them from generalizing to high entropy tasks such as dialogue response generation. Previous work proposes to capture the variability of dialogue responses with a recurrent neural network (RNN)-based conditional variational autoencoder (CVAE). However, the autoregressive computation of the RNN limits the training efficiency. Therefore, we propose the Variational Transformer (VT), a variational self-attentive feed-forward sequence model. The VT combines the parallelizability and global receptive field of the Transformer with the variational nature of the CVAE by incorporating stochastic latent variables into Transformers. We explore two types of the VT: 1) modeling the discourse-level diversity with a global latent variable; and 2) augmenting the Transformer decoder with a sequence of fine-grained latent variables. Then, the proposed models are evaluated on three conversational datasets with both automatic metric and human evaluation. The experimental results show that our models improve standard Transformers and other baselines in terms of diversity, semantic relevance, and human judgment.

Motivation & Objective

• Address the dullness and genericity of deterministic Transformer-based dialogue generation.
• Incorporate stochastic latent variables into Transformers to capture diverse, context-appropriate responses.
• Compare global (discourse-level) and sequential latent-variable designs in dialogue modeling.
• Evaluate on multiple conversational datasets with automatic metrics and human judgments.

Proposed method

• Introduce two VT variants: Global Variational Transformer (GVT) with a global latent variable added to the decoder input, and Sequential Variational Transformer (SVT) with a sequence of latent variables per decoding position.
• Use CVAE-inspired prior and posterior latent-variable modeling within a Transformer framework, leveraging non-causal attention in SVT for latent-variable computation.
• Incorporate KL annealing and a bag-of-words auxiliary loss to mitigate latent-variable vanishing and encourage informative latent representations.
• Train with ELBO objective augmented by SBOW auxiliary loss to encourage latent variables to plan future generation (per position).
• Employ a 4-layer Transformer base with 300 hidden units, 4 attention heads, and 300-d latent variables; reuse MLE pretraining and apply Adam optimization.

Experimental results

Research questions

• RQ1 Can integrating latent variables into Transformer-based dialogue models improve response diversity without sacrificing semantic relevance?
• RQ2 How do global (discourse-level) versus sequential (per-token) latent variables affect generation quality and human judgments?
• RQ3 Do KL-annealing and auxiliary losses stabilize training and preserve useful latent information in VT models?
• RQ4 What are the comparative effects of GVT and SVT on automatic metrics and human evaluation across diverse datasets?

Key findings

• GVT and SVT outperform standard Transformer and CVAE baselines in diversity and human judgments.
• SVT achieves higher semantic relevance as measured by Embedding Similarity (EMB_FT and EMB_BERT) on MojiTalk, with nuanced results on Persona+ED.
• GVT generally reduces reconstruction perplexity (PPL) indicating richer latent information; SVT further improves PPL via sequential latent variables.
• GVT and SVT show improved Dist-1/Dist-2/Dist-3 over baselines, indicating more diverse outputs.
• Human evaluation favors SVT for coherence, emotion, and engagement, while SVT’s per-token latent modeling enhances informativeness in certain datasets.