[Paper Review] The Shmoop Corpus: A Dataset of Stories with Loosely Aligned Summaries

Atef Chaudhury, Makarand Tapaswi+2 more|arXiv (Cornell University)|Dec 30, 2019

Topic Modeling12 references4 citations

TL;DR

The Shmoop Corpus introduces a novel dataset of 7,234 story chapters paired with chronologically aligned, multi-paragraph summaries from Shmoop, enabling improved machine reading comprehension. By leveraging the chronological alignment as a supervisory signal, the authors demonstrate significant gains in Cloze-style question answering and abstractive summarization tasks, showing that structured temporal supervision enhances model performance on long-form narrative understanding.

ABSTRACT

Understanding stories is a challenging reading comprehension problem for machines as it requires reading a large volume of text and following long-range dependencies. In this paper, we introduce the Shmoop Corpus: a dataset of 231 stories that are paired with detailed multi-paragraph summaries for each individual chapter (7,234 chapters), where the summary is chronologically aligned with respect to the story chapter. From the corpus, we construct a set of common NLP tasks, including Cloze-form question answering and a simplified form of abstractive summarization, as benchmarks for reading comprehension on stories. We then show that the chronological alignment provides a strong supervisory signal that learning-based methods can exploit leading to significant improvements on these tasks. We believe that the unique structure of this corpus provides an important foothold towards making machine story comprehension more approachable.

Motivation & Objective

• To address the challenge of machine story comprehension, which requires modeling long-range dependencies across lengthy, complex narratives.
• To provide a more structured and scalable alternative to existing datasets like NarrativeQA, which rely on sparse QA pairs with weak supervision.
• To create a benchmark dataset with fine-grained, chapter-level summaries that mirror human learning aids, enabling better training signals for NLP models.
• To evaluate whether chronological alignment between story and summary paragraphs can serve as a strong supervisory signal for improving downstream reading comprehension tasks.
• To demonstrate that leveraging temporal structure improves performance on Cloze QA and simplified abstractive summarization tasks.

Proposed method

• The corpus was constructed by collecting paired story chapters and Shmoop summaries from the Shmoop website and Project Gutenberg, with manual parsing and chapter segmentation.
• Manual alignment of 5% of the corpus (360 chapters) was performed to verify chronological consistency, using paragraph-level splits based on bullet points (summaries) and line breaks (stories).
• The authors introduced a joint learning framework that optimizes both model parameters and alignment matrices using a modified CCCP algorithm, with dynamic programming enabling efficient computation of optimal alignments.
• A contrastive sampling strategy was used during training to approximate the maximum over all possible alignments, while during validation, the optimal alignment was computed directly.
• The method exploits the chronological structure to compute alignments that preserve temporal order, even allowing for one-to-many and many-to-one mappings common in dialog-heavy or dense narrative passages.
• The framework was evaluated on two tasks: Cloze-form question answering and multiple-choice abstractive summarization, with performance measured using standard NLP metrics.

Experimental results

Research questions

• RQ1 Can chronological alignment between story and summary paragraphs serve as a strong supervisory signal for training machine comprehension models?
• RQ2 How does the use of fine-grained, chapter-level summaries compare to coarse, whole-story summaries in improving reading comprehension benchmarks?
• RQ3 To what extent does modeling long-range dependencies in narrative texts improve performance on Cloze-style question answering and abstractive summarization tasks?
• RQ4 Can a joint optimization framework that learns alignments alongside model parameters lead to better performance than end-to-end training without explicit alignment supervision?
• RQ5 How robust is the chronological structure to variations such as dialog-heavy scenes or dense narrative paragraphs?

Key findings

• The Shmoop Corpus contains 231 stories (145 novels, 62 plays, 24 short stories) with 7,234 chapter-summary pairs, providing a balanced dataset between short-form and long-form narrative data.
• Only 1.2% of manually aligned summary paragraphs deviated from chronological order, indicating strong temporal consistency in the data structure.
• The chronological alignment enabled significant performance gains in both Cloze-form question answering and abstractive summarization tasks, demonstrating its value as a supervisory signal.
• The joint learning framework with dynamic programming-based alignment computation achieved better results than baseline methods, particularly in modeling long-range dependencies across chapters.
• The dataset supports diverse alignment patterns, including one-to-many (e.g., dialog scenes) and many-to-one (e.g., dense narrative paragraphs), showing its adaptability to different narrative structures.
• The results suggest that structured, temporally aligned summaries can significantly improve model generalization and comprehension of complex, long-form narratives.