[Paper Review] Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind

Mo Yu, Wang, Qiujing+9 more|arXiv (Cornell University)|Nov 9, 2022

Topic Modeling4 citations

TL;DR

This paper introduces ToM-in-AMC, the first benchmark for evaluating few-shot meta-learning of theory-of-mind (ToM) in naturalistic narrative understanding using movie scripts. It proposes a novel ToM prompting framework that explicitly models multiple ToM dimensions—beliefs, desires, intentions, and emotions—achieving state-of-the-art performance, yet still lagging over 20% behind human performance, revealing a critical gap in current LLMs' and meta-learning models' ToM capabilities.

ABSTRACT

When reading a story, humans can quickly understand new fictional characters with a few observations, mainly by drawing analogies to fictional and real people they already know. This reflects the few-shot and meta-learning essence of humans' inference of characters' mental states, i.e., theory-of-mind (ToM), which is largely ignored in existing research. We fill this gap with a novel NLP dataset, ToM-in-AMC, the first assessment of machines' meta-learning of ToM in a realistic narrative understanding scenario. Our dataset consists of ~1,000 parsed movie scripts, each corresponding to a few-shot character understanding task that requires models to mimic humans' ability of fast digesting characters with a few starting scenes in a new movie. We propose a novel ToM prompting approach designed to explicitly assess the influence of multiple ToM dimensions. It surpasses existing baseline models, underscoring the significance of modeling multiple ToM dimensions for our task. Our extensive human study verifies that humans are capable of solving our problem by inferring characters' mental states based on their previously seen movies. In comparison, our systems based on either state-of-the-art large language models (GPT-4) or meta-learning algorithms lags >20% behind, highlighting a notable limitation in existing approaches' ToM capabilities.

Motivation & Objective

• To address the lack of realistic, few-shot benchmarks for evaluating machines' meta-learning of theory-of-mind (ToM) in narrative understanding.
• To assess whether models can rapidly infer characters' mental states (beliefs, desires, intentions, emotions) from minimal initial scenes, mimicking human-like few-shot generalization.
• To develop a comprehensive evaluation framework that captures multiple dimensions of ToM, moving beyond static personality classification or dialogue generation.
• To identify and quantify the limitations of current state-of-the-art models (e.g., GPT-4, meta-learning algorithms) in reasoning about complex, dynamic character mental states in realistic story contexts.
• To establish a new standard for evaluating ToM in NLP by leveraging movie scripts as a natural, rich source of narrative reasoning tasks.

Proposed method

• The authors construct a dataset of ~1,000 parsed movie scripts, each transformed into a few-shot character understanding task using only the first few scenes as context.
• Each task requires models to identify the speaker of a dialogue in a scene where all speakers are anonymized, based on minimal prior exposure—mirroring human few-shot inference.
• A novel ToM prompting approach is introduced to explicitly assess and ground reasoning across four ToM dimensions: beliefs, desires, intentions, and emotions.
• The method uses structured prompting to elicit model responses that reason about character mental states, with a focus on analogical reasoning based on prior knowledge of similar characters.
• The dataset is split into meta-training and meta-testing sets, enabling meta-learning evaluation where models must generalize to new movies after seeing only a few initial scenes.
• Performance is evaluated via human studies and comparisons with strong baselines, including GPT-4 with in-context learning and meta-learning models like ProtoNet and LEOPARD.

Experimental results

Research questions

• RQ1 Can models generalize to new narrative characters with only a few initial scenes, as humans do, by leveraging prior knowledge of similar characters?
• RQ2 To what extent do current large language models (e.g., GPT-4) and meta-learning algorithms capture multi-dimensional theory-of-mind reasoning in narrative contexts?
• RQ3 How do different narrative genres and speaker counts affect the difficulty of few-shot character understanding tasks?
• RQ4 What is the performance gap between human reasoning and machine models in inferring complex character mental states from minimal narrative input?
• RQ5 Can a structured ToM prompting framework that explicitly models beliefs, desires, intentions, and emotions significantly improve model performance on few-shot character understanding?

Key findings

• Humans outperform all tested models by over 20% on the ToM-in-AMC benchmark, indicating a significant gap in current models' theory-of-mind capabilities.
• The proposed ToM prompting approach outperforms existing baselines, demonstrating the importance of explicitly modeling multiple ToM dimensions for narrative reasoning.
• GPT-4 with in-context learning (ICL) achieves strong performance (86.5% on 2-speaker scenes), but still lags behind human performance, especially in complex genres like drama and horror.
• Performance varies significantly by genre, with highest scores in adventure (98.4%) and romance (94.7%), and lowest in drama (91.8%) and comedy (86.3%), suggesting genre-dependent reasoning challenges.
• The performance gap widens with increasing numbers of speakers: while GPT-4 ToMPro achieves 90.0% on 1-speaker scenes, it drops to 82.1% on 4-speaker scenes and 90.0% on 5-speaker scenes, indicating scalability issues.
• The study confirms that models often fail to infer global character traits (e.g., technical expertise) from context, as seen in bad cases like The Matrix, where Taylor’s role as a skilled operative is missed.