[Paper Review] Transformer-based Conditional Variational Autoencoder for Controllable Story Generation

Le Fang, Tao Zeng+4 more|arXiv (Cornell University)|Jan 4, 2021

Topic ModelingComputer Science45 references41 citations

TL;DR

The paper revives latent-variable modeling on top of Transformer architectures by embedding a CVAE with GPT-2 as decoder to achieve controllable, long-form story generation conditioned on prompts, while maintaining strong generation quality.

ABSTRACT

We investigate large-scale latent variable models (LVMs) for neural story generation -- an under-explored application for open-domain long text -- with objectives in two threads: generation effectiveness and controllability. LVMs, especially the variational autoencoder (VAE), have achieved both effective and controllable generation through exploiting flexible distributional latent representations. Recently, Transformers and its variants have achieved remarkable effectiveness without explicit latent representation learning, thus lack satisfying controllability in generation. In this paper, we advocate to revive latent variable modeling, essentially the power of representation learning, in the era of Transformers to enhance controllability without hurting state-of-the-art generation effectiveness. Specifically, we integrate latent representation vectors with a Transformer-based pre-trained architecture to build conditional variational autoencoder (CVAE). Model components such as encoder, decoder and the variational posterior are all built on top of pre-trained language models -- GPT2 specifically in this paper. Experiments demonstrate state-of-the-art conditional generation ability of our model, as well as its excellent representation learning capability and controllability.

Motivation & Objective

• Motivate and enable controllable, open-domain long-text generation using latent variable models.
• Integrate a CVAE with a Transformer-based pre-trained backbone to improve controllability without sacrificing generation quality.
• Study how latent representations can guide long-form storytelling conditioned on prompts.

Proposed method

• Build a Transformer-based CVAE with a GPT-2 decoder and unmasked bidirectional encoder initialized from GPT-2 layers.
• Represent the latent code z as an isotropic Gaussian with learnable mean and log-variance.
• Inject latent code into the decoder via multiple latent-injection strategies (input addition, pseudo self-attention, and a softmax-head projection).
• Train with an ELBO objective for CVAE, using a cyclic annealing schedule to mitigate posterior collapse.
• Use an attention-average block to summarize variable-length encoder outputs into a single latent vector.

Experimental results

Research questions

• RQ1 Can Transformer-based CVAEs achieve competitive or superior conditional story generation quality compared to non-latent or plain transformer baselines?
• RQ2 Does learning a distributional latent space improve controllability and representation of prompts in long-form storytelling?
• RQ3 Which latent-code injection method (among three proposed) yields best generation performance and controllability for long text?
• RQ4 How well does the model scale to open-domain long-text datasets like WritingPrompts and WikiPlots?

Key findings

• The Transformer-based CVAE achieves generally better or comparable perplexity and ROUGE scores than baselines on WritingPrompts and WikiPlots, indicating strong conditional generation performance.
• Learned latent representations enable controllability; latent codes can steer generation when combined with prompts.
• Among latent-injection methods, CVAE-① and CVAE-② perform comparably well, with CVAE-③ generally less effective in practice for long text.
• Pre-trained backbone (GPT-2) based models (PSA, FIST, CVAE) show strong performance gains over fusion-based baselines in open-domain long-text tasks.
• Qualitative analysis shows latent codes cluster prompts meaningfully in latent space and can guide generation toward prompt-consistent content.
• A latent-variable Transformer model demonstrates controllability in long-form storytelling without compromising generation quality.