[Paper Review] Colin: A Multimodal Human-AI Co-Creation Storytelling System To Support Children's Multi-Level Narrative Skills

Lyumanshan Ye, Jiandong Jiang+8 more|arXiv (Cornell University)|May 10, 2024

Digital Storytelling and EducationHealth Professions3 citations

TL;DR

Storypark is a multimodal child-AI co-creation storytelling system that uses a question-scaffolding feedback framework with large language models (LLMs) to enhance children’s narrative skills, core value understanding, and story comprehension. It integrates voice, drawing, and LLM-driven dialogue to support real-time, adaptive story co-creation, significantly improving learning outcomes and user engagement in children aged 3–8.

ABSTRACT

Children develop narrative skills by understanding and actively building connections between elements, image text matching, and consequences. However, it is challenging for children to clearly grasp these multi level links only through explanations of text or the facilitator's speech. To address this, we developed Colin, an interactive storytelling tool that supports children's multi level narrative skills through both voice and visual modalities. In the generation stage, Colin supports the facilitator to define and review the generated text and image content freely. In the understanding stage, a question feedback model helps children understand multi level connections while co creating stories with Colin. In the building phase, Colin actively encourages children to create connections between elements through drawing and speaking. A user study with 20 participants evaluated Colin by measuring children's engagement, understanding of cause and effect relationships, and the quality of their new story creations. Our results demonstrate that Colin significantly enhances the development of children's narrative skills across multiple levels.

Motivation & Objective

• To address the challenge of facilitators lacking real-time, personalized support in interactive storytelling due to high cognitive and creative demands.
• To design a system that enables children to co-create stories with AI by integrating voice, drawing, and LLM-driven dialogue in a child-centered, multimodal interface.
• To evaluate whether AI-supported interactive storytelling enhances children’s understanding of story core values, narrative generalization, and transfer of learning.
• To explore how LLMs can be customized and constrained to support age-appropriate, educationally meaningful story development while minimizing hallucinations and logical errors.

Proposed method

• The system employs a three-phase interactive loop: (1) posing open-ended questions to elicit children’s narrative contributions, (2) using LLMs to generate contextually coherent story expansions based on children’s input and a predefined story framework, and (3) rendering story continuations with multimodal outputs including voice, text, and AI-generated drawings.
• A question-scaffolding feedback framework is implemented to guide children’s narrative contributions using prompts that encourage elaboration, inference, and value-based interpretation.
• LLM outputs are constrained via prompt engineering, conversation history tracking, and input filtering (e.g., removing dialogue from keyword extraction) to maintain narrative consistency and reduce deviation from the story outline.
• The system incorporates visual co-creation by generating drawings based on story content, providing visual cues to support children’s comprehension and creative expression.
• Validation layers and feedback loops are applied to mitigate LLM hallucinations, especially in content involving real-world logic or domain-specific knowledge.
• The system is designed with age-specific interaction modes, adjusting question complexity and learning focus based on developmental stages (3–6, 6–8, 8+ years old).

Figure 1. Child interaction process

Experimental results

Research questions

• RQ1 How can an LLM-powered system support children in co-creating stories while maintaining narrative coherence and educational alignment?
• RQ2 To what extent does child-AI collaborative storytelling improve children’s understanding of story core values, generalization, and transfer of learning?
• RQ3 How do multimodal interactions (voice, drawing, text) influence children’s engagement and narrative development in AI-assisted storytelling?
• RQ4 What design strategies are effective in minimizing LLM hallucinations and ensuring educational accuracy in child-AI storytelling systems?

Key findings

• Storypark significantly improved children’s learning outcomes in understanding story key ideas, generalization, and transfer of narrative concepts, as measured through post-interaction assessments.
• Children demonstrated high engagement and expressed strong willingness to use the system again, indicating a positive user experience and perceived value as a collaborative storytelling partner.
• The integration of drawing features enhanced children’s narrative expression and provided visual scaffolding that supported comprehension and creative output.
• Despite challenges with LLM hallucinations—such as generating implausible story continuations (e.g., making herbal jelly from turtle shells)—these did not severely disrupt educational goals in a value-based learning context.
• The system’s adaptive question-scaffolding framework effectively guided children’s contributions and supported story development without overwhelming or derailing the narrative.
• User studies with 20 child participants confirmed the system’s usability and effectiveness in supporting multi-level narrative skills across different developmental stages.

Figure 2. User Interface of StoryPark