[Paper Review] TVRecap: A Dataset for Generating Stories with Character Descriptions.

Mingda Chen, Kevin Gimpel|arXiv (Cornell University)|Sep 18, 2021

Topic Modeling35 references4 citations

TL;DR

TVRecap is a large-scale dataset of 26,000 professionally written TV episode recaps, created from fan websites, that requires generating detailed, character-rich stories from brief summaries and extensive character documents. A hierarchical neural model with oracle content selectors achieves the best performance on automatic metrics, demonstrating the dataset’s value for constrained story generation and abstractive summarization research.

ABSTRACT

We introduce TVRecap, a story generation dataset that requires generating detailed TV show episode recaps from a brief summary and a set of documents describing the characters involved. Unlike other story generation datasets, TVRecap contains stories that are authored by professional screenwriters and that feature complex interactions among multiple characters. Generating stories in TVRecap requires drawing relevant information from the lengthy provided documents about characters based on the brief summary. In addition, by swapping the input and output, TVRecap can serve as a challenging testbed for abstractive summarization. We create TVRecap from fan-contributed websites, which allows us to collect 26k episode recaps with 1868.7 tokens on average. Empirically, we take a hierarchical story generation approach and find that the neural model that uses oracle content selectors for character descriptions demonstrates the best performance on automatic metrics, showing the potential of our dataset to inspire future research on story generation with constraints. Qualitative analysis shows that the best-performing model sometimes generates content that is unfaithful to the short summaries, suggesting promising directions for future work.

Motivation & Objective

• To create a large-scale, professionally authored story generation dataset featuring complex character interactions and detailed descriptions.
• To address the challenge of generating coherent, contextually accurate stories by drawing on extensive character documents and brief summaries.
• To provide a benchmark for abstractive summarization by reversing input and output roles in the story generation task.
• To inspire research on neural models that can effectively select and integrate relevant information from long character descriptions.
• To evaluate the fidelity and quality of generated stories in relation to source summaries and character details.

Proposed method

• Constructed TVRecap from fan-contributed websites, collecting 26,000 episode recaps with an average length of 1,868.7 tokens.
• Designed a hierarchical story generation framework that processes summaries and character documents to generate detailed recaps.
• Implemented an oracle content selector mechanism to identify and incorporate the most relevant information from character documents.
• Evaluated models using automatic metrics, comparing performance across different content selection strategies.
• Repurposed the dataset for abstractive summarization by treating recaps as inputs and summaries as outputs.
• Conducted qualitative analysis to assess factual consistency between generated content and input summaries.

Experimental results

Research questions

• RQ1 Can neural models effectively generate detailed, coherent TV episode recaps using only brief summaries and extensive character descriptions?
• RQ2 How does the use of oracle content selectors improve story generation performance compared to baseline methods?
• RQ3 To what extent do generated stories remain faithful to the original short summaries, despite complex character interactions?
• RQ4 Can TVRecap serve as a viable benchmark for abstractive summarization tasks?
• RQ5 What are the key failure modes in story generation when models diverge from input summaries?

Key findings

• The hierarchical neural model with oracle content selectors achieved the highest performance on automatic evaluation metrics.
• The dataset contains 26,000 professionally written episode recaps, with an average length of 1,868.7 tokens per recap.
• The best-performing model occasionally generated content that was inconsistent with the input summaries, indicating a need for improved faithfulness.
• TVRecap serves as a valid and challenging benchmark for abstractive summarization when inputs and outputs are reversed.
• Qualitative analysis revealed that model outputs sometimes introduced factual inaccuracies or diverged from the original summary.
• The results suggest that integrating long-form character descriptions effectively remains a key challenge in story generation.