[Paper Review] Consistency and Coherency Enhanced Story Generation

Wei Wang, Piji Li+1 more|arXiv (Cornell University)|Oct 17, 2020

Topic Modeling29 references4 citations

TL;DR

This paper proposes a two-stage story generation framework that enhances consistency and coherency by first constructing a structured story outline and then expanding it into a full narrative. By incorporating coreference supervision and a discourse relation classification head, the model significantly reduces coreference errors and improves sentence-level coherence, outperforming GPT2-based baselines in both automatic metrics and human evaluation.

ABSTRACT

Story generation is a challenging task, which demands to maintain consistency of the plots and characters throughout the story. Previous works have shown that GPT2, a large-scale language model, has achieved good performance on story generation. However, we observe that several serious issues still exist in the stories generated by GPT2 which can be categorized into two folds: consistency and coherency. In terms of consistency, on one hand, GPT2 cannot guarantee the consistency of the plots explicitly. On the other hand, the generated stories usually contain coreference errors. In terms of coherency, GPT2 does not take account of the discourse relations between sentences of stories directly. To enhance the consistency and coherency of the generated stories, we propose a two-stage generation framework, where the first stage is to organize the story outline which depicts the story plots and events, and the second stage is to expand the outline into a complete story. Therefore the plots consistency can be controlled and guaranteed explicitly. In addition, coreference supervision signals are incorporated to reduce coreference errors and improve the coreference consistency. Moreover, we design an auxiliary task of discourse relation modeling to improve the coherency of the generated stories. Experimental results on a story dataset show that our model outperforms the baseline approaches in terms of both automatic metrics and human evaluation.

Motivation & Objective

• To address persistent issues in GPT2-generated stories, including plot inconsistency and coreference errors.
• To improve discourse coherency by explicitly modeling discourse relations between sentences.
• To develop a two-stage framework that ensures plot consistency through structured story outlining.
• To integrate auxiliary supervision for coreference resolution and discourse structure to enhance narrative quality.
• To evaluate the model on automatic metrics and human evaluation, demonstrating superiority over GPT2-based baselines.

Proposed method

• The model uses a two-stage approach: first generating a story outline that captures plot events and character states, then expanding it into a full narrative.
• Coreference supervision is applied during training to reduce ambiguity in pronoun and noun reference, improving coreference consistency.
• A discourse relation classification head is introduced as an auxiliary task to predict relations (e.g., contrast, cause, condition) between sentence pairs.
• The loss function combines language modeling loss with a weighted loss for discourse relation prediction, with λ₁ = 0.1 found optimal.
• The model leverages a pre-trained Transformer-based autoregressive language model (TLM) for both stages, fine-tuned with structured supervision.
• Top-k sampling is used during decoding, but the structured generation process reduces the impact of random token generation on coherence.

Experimental results

Research questions

• RQ1 Can a two-stage generation framework improve plot consistency in story generation compared to end-to-end autoregressive models?
• RQ2 How does coreference supervision affect the reduction of coreference errors in generated stories?
• RQ3 To what extent does discourse relation modeling enhance sentence-level coherency in long-form narratives?
• RQ4 What is the optimal trade-off between language modeling and auxiliary discourse relation prediction in story generation?
• RQ5 Does integrating structured story outlining lead to more coherent and consistent narratives than direct autoregressive generation?

Key findings

• The proposed model outperforms GPT2 and GPT2P in both automatic metrics and human evaluation, demonstrating improved story quality.
• The model achieves the highest percentage of correctly predicted discourse relations and the lowest proportion of unknown relations, indicating better discourse coherency.
• Coreference supervision significantly reduces coreference errors, with fewer ambiguous or incorrect pronoun references in the generated stories.
• The two-stage framework ensures better plot consistency, as evidenced by fewer contradictions in character states and event sequences.
• The optimal weight λ₁ = 0.1 for discourse loss was found to balance the auxiliary task with the main language modeling objective without degrading performance.
• Case studies show that the model generates more coherent, consistent, and logically structured stories compared to GPT2P, especially in maintaining character and plot continuity.