[Paper Review] Exploring Story Generation with Multi-task Objectives in Variational Autoencoders

Zhuohan Xie, Trevor Cohn+1 more|arXiv (Cornell University)|Nov 15, 2021

Topic Modeling28 references4 citations

TL;DR

This paper proposes a domain-specific variational autoencoder (VAE) that combines BERT and GPT-2 to improve story generation by introducing two auxiliary objectives: story topic prediction and discourse relation discrimination. The enhanced VAE learns more informative latent variables, achieving a better quality-diversity trade-off than GPT-2 and vanilla VAE across multiple datasets.

ABSTRACT

GPT-2 has been frequently adapted in story generation models as it provides powerful generative capability. However, it still fails to generate consistent stories and lacks diversity. Current story generation models leverage additional information such as plots or commonsense into GPT-2 to guide the generation process. These approaches focus on improving generation quality of stories while our work look at both quality and diversity. We explore combining BERT and GPT-2 to build a variational autoencoder (VAE), and extend it by adding additional objectives to learn global features such as story topic and discourse relations. Our evaluations show our enhanced VAE can provide better quality and diversity trade off, generate less repetitive story content and learn a more informative latent variable.

Motivation & Objective

• To address the limitations of GPT-2 in story generation, particularly low diversity and inconsistency in storylines.
• To improve the informativeness of latent variables in VAEs for story generation by incorporating global story features.
• To explore multi-task learning objectives that guide the VAE to capture story topics and discourse quality.
• To develop a domain-specific VAE that outperforms GPT-2 and vanilla VAE in quality-diversity trade-off on story generation tasks.

Proposed method

• Construct a VAE using BERT as the encoder and GPT-2 as the decoder, pre-trained on domain-specific story data.
• Introduce a multi-task learning framework where the latent variable is jointly optimized for story topic classification and original vs. corrupted story discrimination.
• Use a negative sampling strategy to simulate common generation errors, enabling the model to learn discourse-level coherence.
• Apply top-p sampling with varying p-values to evaluate generation quality and diversity across multiple hyperparameter settings.
• Train the VAE using a combination of reconstruction loss, KL divergence, and two auxiliary objectives to regularize the latent space.
• Fine-tune the VAE on story datasets and evaluate using metrics like negative Corpus-BLEU, self-BLEU, and ROC AUC for discourse quality.

Experimental results

Research questions

• RQ1 Can a VAE built from BERT and GPT-2 learn more informative latent representations for story generation than standard GPT-2?
• RQ2 Does adding auxiliary objectives for topic modeling and discourse quality improve the quality-diversity trade-off in story generation?
• RQ3 Can the latent space of a VAE be effectively regularized to capture global story features like topic and coherence, rather than local patterns?
• RQ4 How does the proposed multi-task VAE compare to GPT-2 and vanilla VAE in terms of generation quality and diversity across different datasets?

Key findings

• The enhanced VAE (VAE + t, VAE + d, VAE + td) achieves a better quality-diversity trade-off than GPT-2 and vanilla VAE across all tested datasets, as shown by the trade-off curves in Figure 3.
• The VAE with topic modeling (VAE + t) and discourse discrimination (VAE + d) both outperformed the vanilla VAE, with the combined objective (VAE + td) showing the strongest performance on most datasets.
• The model achieved an AUC of 0.75 for discourse discrimination and 0.25 for topic prediction, indicating effective learning of global story features.
• Self-BLEU scores for 4-gram repetition were significantly lower for the enhanced VAEs at low p-values (e.g., p=0.4), showing reduced repetition and higher diversity.
• At high p-values (e.g., p=1.0), all models including GPT-2 achieved self-BLEU scores close to human-written stories (0.021), but the enhanced VAE maintained better quality at lower p-values.
• The AE baseline produced incoherent stories despite low repetition, confirming that low self-BLEU does not imply high quality, and highlighting the advantage of the proposed VAE in balancing quality and diversity.