[Paper Review] "Let Your Characters Tell Their Story": A Dataset for Character-Centric Narrative Understanding

Faeze Brahman, Meng Huang+4 more|arXiv (Cornell University)|Sep 12, 2021

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces LiSCU, a new dataset of literary summaries paired with character-centric descriptions to advance character-centric narrative understanding in NLP. It proposes two novel tasks—Character Identification and Character Description Generation—demonstrating that while existing models can generate fluent descriptions, they often lack faithfulness to source summaries and struggle with long-context comprehension, highlighting the need for improved models in narrative understanding.

ABSTRACT

When reading a literary piece, readers often make inferences about various characters' roles, personalities, relationships, intents, actions, etc. While humans can readily draw upon their past experiences to build such a character-centric view of the narrative, understanding characters in narratives can be a challenging task for machines. To encourage research in this field of character-centric narrative understanding, we present LiSCU -- a new dataset of literary pieces and their summaries paired with descriptions of characters that appear in them. We also introduce two new tasks on LiSCU: Character Identification and Character Description Generation. Our experiments with several pre-trained language models adapted for these tasks demonstrate that there is a need for better models of narrative comprehension.

Motivation & Objective

• To address the challenge of machine understanding of fictional characters in narratives, which requires abstraction beyond simple extraction.
• To develop a new dataset, LiSCU, that pairs literary summaries with in-depth character descriptions from the character’s perspective.
• To introduce two novel NLP tasks: Character Identification (from anonymized descriptions) and Character Description Generation (abstractive, perspective-based analysis).
• To evaluate modern pre-trained language models on these tasks and identify limitations in faithfulness, abstraction, and long-context reasoning.
• To analyze gender bias in the dataset and model outputs, finding that while male characters are more frequent, descriptions of male and female characters are similarly detailed, suggesting no disparity in analytical treatment.

Proposed method

• The LiSCU dataset comprises literary summaries and corresponding character descriptions written from the character’s perspective, emphasizing roles, traits, relationships, and motivations.
• Character Description Generation is framed as an abstractive generation task, distinct from summarization, requiring inference of attributes and roles not explicitly stated in the text.
• Character Identification is formulated as a classification task, where models must identify the correct character name from anonymized descriptions given a summary.
• The dataset includes long summaries (average 1022 tokens), posing challenges for long-context modeling in NLP systems.
• Human evaluations were conducted via Amazon Mechanical Turk to assess quality and model outputs, with paid, time-based compensation to ensure fairness.
• A bias analysis was performed on gender representation, including character frequency, mention counts in summaries, and description length, to evaluate potential disparities.

Experimental results

Research questions

• RQ1 To what extent can pre-trained language models generate character descriptions that are faithful to the source summaries and capture salient attributes and roles?
• RQ2 How well do models perform in identifying the correct character from anonymized descriptions when given a narrative summary?
• RQ3 Can models effectively handle long narrative contexts (average 1022 tokens) to generate coherent and accurate character descriptions?
• RQ4 Is there a gender bias in the dataset or in model outputs, particularly in the treatment of male versus female characters?
• RQ5 How do model-generated descriptions compare to human-annotated descriptions in terms of abstraction, coherence, and factual consistency?

Key findings

• Existing pre-trained language models can generate fluent and logically consistent character descriptions, but they often fail to remain faithful to the source summaries, omitting or distorting key details.
• The average human performance on the Character Identification task is significantly higher than that of current models, indicating a gap in model capability for character inference.
• Models struggle with long-context narratives, as the average summary length (1022 tokens) exceeds the comfort zone of most existing NLP systems.
• Although male characters are more frequent in the dataset (66%), the average description length for male and female characters is nearly identical (203 vs. 200 tokens), suggesting no bias in descriptive depth.
• The average number of mentions of male and female characters in summaries is very similar (32.1 vs. 31.7), indicating that female characters, though fewer, are proportionally prominent in narrative development.
• Despite the imbalance in character frequency, there is no significant difference in the length or sentence count of descriptions between male and female characters, indicating that the human-annotated descriptions do not exhibit gender-based analytical bias.