[Paper Review] CREATIVESUMM: Shared Task on Automatic Summarization for Creative Writing

Divyansh Agarwal, Alexander R. Fabbri+8 more|arXiv (Cornell University)|Nov 10, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces CreativeSumm, a shared task at COLING 2022 focused on automatic summarization of creative writing across four domains: book chapters, movie scripts, primetime TV scripts, and daytime soap opera scripts. It presents curated datasets, evaluation metrics, and analyzes 18 submissions, revealing challenges in handling complex narrative structures and genre-specific styles, with ROUGE scores ranging from 0.29 to 0.39 and low performance on faithfulness and coherence metrics.

ABSTRACT

This paper introduces the shared task of summarizing documents in several creative domains, namely literary texts, movie scripts, and television scripts. Summarizing these creative documents requires making complex literary interpretations, as well as understanding non-trivial temporal dependencies in texts containing varied styles of plot development and narrative structure. This poses unique challenges and is yet underexplored for text summarization systems. In this shared task, we introduce four sub-tasks and their corresponding datasets, focusing on summarizing books, movie scripts, primetime television scripts, and daytime soap opera scripts. We detail the process of curating these datasets for the task, as well as the metrics used for the evaluation of the submissions. As part of the CREATIVESUMM workshop at COLING 2022, the shared task attracted 18 submissions in total. We discuss the submissions and the baselines for each sub-task in this paper, along with directions for facilitating future work in the field.

Motivation & Objective

• To advance research in automatic text summarization beyond news and technical domains by focusing on creative writing.
• To address the underexplored challenge of summarizing long-form, structurally complex creative texts with non-linear plot development and character interdependencies.
• To provide standardized datasets and evaluation protocols for four distinct creative genres: literary books, movie scripts, primetime TV, and daytime soap operas.
• To evaluate state-of-the-art models on these novel tasks and identify limitations in faithfulness, coherence, and abstractive capability.
• To stimulate future work by expanding datasets and improving evaluation metrics for creative text summarization.

Proposed method

• Curated four distinct sub-tasks using existing datasets: BookSumm-chapters (from Project Gutenberg and study guides), Scriptbase (movie scripts with user summaries), and SummScreen (primetime and daytime TV show transcripts).
• Combined and filtered datasets to ensure non-overlapping book titles and remove unreliable or non-narrative content (e.g., plays).
• Established evaluation protocols using ROUGE, LEXICAL, and faithfulness metrics (SummaC, LP) to assess factual consistency and content coverage.
• Provided baseline models including LED and BART-based architectures fine-tuned on each sub-task, with ablation and hyperparameter tuning.
• Hosted the shared task at COLING 2022, collecting 18 submissions across the four sub-tasks for comparative analysis.
• Used GitHub repository to share data, instructions, and evaluation scripts for reproducibility and future extension.

Experimental results

Research questions

• RQ1 How do existing sequence-to-sequence models perform on summarizing long-form creative texts with complex narrative structures?
• RQ2 What are the key differences in summarization performance across genres such as literary fiction, movie scripts, and primetime vs. daytime TV shows?
• RQ3 To what extent do current models preserve factual consistency and narrative coherence in summaries of creative content?
• RQ4 How do different evaluation metrics (ROUGE, SummaC, LP) correlate with human judgments of quality in creative text summarization?
• RQ5 What are the main failure modes of current models when summarizing creative texts with non-traditional discourse structures?

Key findings

• ROUGE-1 scores ranged from 0.29 to 0.39 across sub-tasks, with the highest performance on the Scriptbase (movie scripts) task.
• All systems showed low performance on faithfulness metrics: LP scores were below 0.5, and SummaC scores were consistently low, indicating poor factual consistency.
• The best-performing system for SummScreenFD (daytime soaps) generated shorter, more abstractive summaries, suggesting genre-specific summarization strategies are effective.
• LED-based baselines performed poorly on SummScreenTMS (primetime TV), producing repetitive, extractive outputs likely due to training optimization issues.
• Performance differences between SummScreenFD and TMS (0.29 vs. 0.39 ROUGE-1) highlight significant data distribution differences even within the same domain.
• Despite high ROUGE scores on movie scripts, systems struggled with coherence and factual accuracy, indicating a gap between automatic metrics and human evaluation.