[Paper Review] Large Language Models Meet Harry Potter: A Bilingual Dataset for Aligning Dialogue Agents with Characters

Nuo Chen, Yan Wang+6 more|arXiv (Cornell University)|Nov 13, 2022

Topic Modeling4 citations

TL;DR

This paper introduces the Harry Potter Dialogue (HPD) dataset, a bilingual (English/Chinese) benchmark with dynamically annotated scenes, character attributes, and evolving relationships to improve dialogue agent alignment with fictional characters. Despite strong LLM performance, models fail to generate contextually accurate, character-consistent responses, highlighting the need for better alignment techniques in narrative-driven dialogue systems.

ABSTRACT

In recent years, Dialogue-style Large Language Models (LLMs) such as ChatGPT and GPT4 have demonstrated immense potential in constructing open-domain dialogue agents. However, aligning these agents with specific characters or individuals remains a considerable challenge due to the complexities of character representation and the lack of comprehensive annotations. In this paper, we introduce the Harry Potter Dialogue (HPD) dataset, designed to advance the study of dialogue agents and character alignment. The dataset encompasses all dialogue sessions (in both English and Chinese) from the Harry Potter series and is annotated with vital background information, including dialogue scenes, speakers, character relationships, and attributes. These extensive annotations may empower LLMs to unlock character-driven dialogue capabilities. Furthermore, it can serve as a universal benchmark for evaluating how well can a LLM aligning with a specific character. We benchmark LLMs on HPD using both fine-tuning and in-context learning settings. Evaluation results reveal that although there is substantial room for improvement in generating high-quality, character-aligned responses, the proposed dataset is valuable in guiding models toward responses that better align with the character of Harry Potter.

Motivation & Objective

• To address the challenge of aligning dialogue agents with specific fictional characters in story-driven contexts.
• To overcome limitations in existing datasets that lack dynamic, time-evolving character attributes and relationships.
• To create a benchmark for evaluating both generation-based and retrieval-based dialogue agents in character-aligned settings.
• To investigate how LLMs can be guided to produce responses consistent with character development, relationships, and story context.
• To provide a resource for advancing personalized dialogue systems in virtual environments through fine-grained narrative understanding.

Proposed method

• The HPD dataset collects all dialogue sessions from the Harry Potter novels in both English and Chinese, totaling 1,042 training and 149 test sessions.
• Each dialogue is annotated with scene context, speaker attributes, dynamic character relationships (e.g., friendship, rivalry), and temporal evolution of these relations.
• Annotations include time-stamped changes in character states, such as shifting friendships or emotional states, to reflect story progression.
• The dataset supports both few-shot in-context learning and fine-tuning evaluation protocols for LLMs.
• A test set with 1–3 positive and 9 negative responses per sample enables rigorous evaluation of response relevance and character consistency.
• The dataset is released with a script to extract raw dialogue using line and page numbers, ensuring reproducibility and ethical use.

Experimental results

Research questions

• RQ1 How well can LLMs generate responses that align with a character’s personality, relationships, and evolving story context?
• RQ2 To what extent do current LLMs hallucinate or contradict established narrative facts when simulating character behavior?
• RQ3 Can dynamic annotations of character attributes and relationships significantly improve LLM alignment with fictional characters?
• RQ4 What are the limitations of in-context learning and fine-tuning in achieving narrative-consistent dialogue generation?
• RQ5 How do relationship dynamics (e.g., temporary estrangement) affect LLM response generation, and can models capture such nuances?

Key findings

• LLMs like ChatGPT and GPT-4 frequently generate responses that contradict the story context, such as having Harry and Ron reconcile during a period of mutual avoidance.
• Even with extensive context prompts, models fail to represent Harry’s reluctance to see Ron after a conflict, instead producing responses consistent with their friendship.
• Models often generate responses starting with 'Yes, I was...', which misrepresent Harry’s intent to downplay his ability to speak to snakes.
• The case study in Book 4, Chapter 19 shows that models fail to capture Harry’s fear and desire to conceal his ability, instead producing overly candid or truthful replies.
• Despite being trained on the Harry Potter universe, LLMs struggle with nuanced character behavior, indicating that next-word prediction alone is insufficient for deep narrative alignment.
• The HPD benchmark reveals significant gaps in current LLM capabilities for character-aligned dialogue, especially in handling dynamic relationships and temporal consistency.