[Paper Review] Designing Human and Generative AI Collaboration

Kartik Hosanagar, Daehwan Ahn|arXiv (Cornell University)|Dec 14, 2024

Big Data and Business IntelligenceBusiness, Management and Accounting3 citations

TL;DR

This study investigates how different collaboration designs between humans and generative AI affect creative writing outcomes. Through a controlled experiment, it finds that active human involvement in early creative stages significantly improves story quality, interestingness, and user satisfaction, while reducing over-reliance on AI and preserving creative diversity.

ABSTRACT

We examined the effectiveness of various human-AI collaboration designs on creative work. Through a human subjects experiment set in the context of creative writing, we found that while AI assistance improved productivity across all models, collaboration design significantly influenced output quality, user satisfaction, and content characteristics. Models incorporating human creative input delivered higher content interestingness and overall quality as well as greater task performer satisfaction compared to conditions where humans were limited to confirming AI's output. Increased AI involvement encouraged creators to explore beyond personal experience but also led to lower aggregate diversity in stories and genres among participants. However, this effect was mitigated through human participation in early creative tasks. These findings underscore the importance of preserving the human creative role to ensure quality, satisfaction, and creative diversity in human-AI collaboration.

Motivation & Objective

• To investigate how varying levels of human and AI involvement in creative writing tasks influence output quality and user experience.
• To examine the impact of collaboration design on creative diversity, including genre and thematic variation in generated stories.
• To assess how human participation in early-stage ideation affects the final quality and originality of AI-assisted creative work.
• To identify design principles that preserve human creative agency while leveraging AI productivity gains.

Proposed method

• Conducted a human subjects experiment in a creative writing context with controlled collaboration conditions.
• Compared AI collaboration models where humans either generated initial ideas or only confirmed AI-generated content.
• Used standardized metrics to evaluate story quality, interestingness, and user satisfaction.
• Collected and analyzed narrative outputs for diversity in genres, themes, and narrative structures.
• Employed qualitative feedback and quantitative ratings to assess user experience and perceived satisfaction.
• Applied supplementary analysis to examine the relationship between human input timing and creative outcomes.

Experimental results

Research questions

• RQ1 How does the timing and extent of human involvement in the creative process affect the quality and interestingness of AI-assisted stories?
• RQ2 What is the impact of AI-only generation versus human-AI co-creation on narrative diversity and originality?
• RQ3 How does collaboration design influence user satisfaction and perceived agency in AI-assisted creative tasks?
• RQ4 To what extent does increased AI involvement reduce the diversity of genres and themes in user-generated stories?

Key findings

• Collaboration models that required human input in early ideation stages produced significantly higher-rated stories in terms of interestingness and overall quality.
• Participants reported greater satisfaction when they contributed creatively rather than merely confirming AI output.
• Conditions with limited human input led to lower aggregate diversity in genres and themes across stories.
• Increased AI involvement encouraged exploration beyond personal experience but at the cost of reduced narrative diversity.
• Human participation in early stages mitigated the decline in creative diversity typically associated with high AI involvement.
• AI assistance improved productivity across all models, but quality and satisfaction were strongly dependent on collaboration design.