[Paper Review] Hierarchical Neural Story Generation

Angela Fan, Mike Lewis+1 more|arXiv (Cornell University)|May 13, 2018

Topic Modeling3 references152 citations

TL;DR

The paper introduces a hierarchical approach to story generation using a two-stage process (premise then story) with a fusion mechanism and gated multi-scale self-attention to improve coherence and prompt relevance, evaluated on a large WritingPrompts dataset.

ABSTRACT

We explore story generation: creative systems that can build coherent and fluent passages of text about a topic. We collect a large dataset of 300K human-written stories paired with writing prompts from an online forum. Our dataset enables hierarchical story generation, where the model first generates a premise, and then transforms it into a passage of text. We gain further improvements with a novel form of model fusion that improves the relevance of the story to the prompt, and adding a new gated multi-scale self-attention mechanism to model long-range context. Experiments show large improvements over strong baselines on both automated and human evaluations. Human judges prefer stories generated by our approach to those from a strong non-hierarchical model by a factor of two to one.

Motivation & Objective

• Motivate and tackle open-ended story generation by enforcing high-level planning via hierarchy.
• Create a large-scale prompt-story dataset to study long-range coherence in narratives.
• Develop model innovations (fusion with a pretrained model, gated multi-scale self-attention) to maintain relevance to prompts.
• Evaluate improvements with both automated metrics and human judgments on coherence and prompt adherence.

Proposed method

• Construct a two-stage generation: first generate a story prompt (premise) with a convolutional language model, then generate the story conditioned on this prompt using a Conv seq2seq model.
• Adopt a convolutional encoder-decoder architecture to enable parallel processing of long texts.
• Introduce gated multi-scale self-attention to model unbounded context with different time scales and gating to select past information.
• Apply a fusion mechanism (cold fusion-like) to combine a pretrained seq2seq with a second seq2seq model to improve conditioning on the prompt.
• Evaluate perplexity and prompt relevance, plus human judgments, using top-k sampling for generation and a prompt-language model for prompt creation.

Experimental results

Research questions

• RQ1 Can hierarchical generation (premise first, then story) improve coherence and topical adherence to prompts in long-form narrative generation?
• RQ2 Do fusion with a pretrained model and gated multi-scale self-attention improve prompt relevance and long-range coherence compared to non-hierarchical baselines?
• RQ3 How does the open-ended prompt-story setting perform under human evaluation versus automated metrics?
• RQ4 What are the effects of the proposed attention mechanisms on modeling long-range dependencies in stories?

Key findings

• Hierarchical generation significantly improves human preference for stories over a non-hierarchical baseline (67.32% vs 32.68%).
• Gated multi-scale self-attention and the new attention mechanism substantially reduce perplexity on the WritingPrompts dataset (e.g., from 45.27 to 37.94 in test perplexity with gating).
• Model fusion (training a second seq2seq model on top of a pretrained one) substantially improves the likelihood that generated stories remain connected to the prompt, outperforming an ensemble with fewer parameters.
• Fusion enables learning dependencies between prompt and story that standard seq2seq models struggle to capture, with improved prompt-story pairing accuracy in human evaluation.
• The fusion model can match nearest-neighbor performance in link quality between prompt and story while enabling unlimited generation beyond training instances.