[Paper Review] Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers

Daphne Ippolito, Ann Yuan+2 more|arXiv (Cornell University)|Nov 9, 2022

Topic Modeling49 citations

TL;DR

The paper reports on a long-term study with 13 professional writers using Wordcraft, an AI-assisted writing editor, to explore workflows, limitations, and the role of AI in creative writing. It highlights challenges in maintaining voice, over-reliance on tropes, and user interface importance for productive collaboration.

ABSTRACT

Recent developments in natural language generation (NLG) using neural language models have brought us closer than ever to the goal of building AI-powered creative writing tools. However, most prior work on human-AI collaboration in the creative writing domain has evaluated new systems with amateur writers, typically in contrived user studies of limited scope. In this work, we commissioned 13 professional, published writers from a diverse set of creative writing backgrounds to craft stories using Wordcraft, a text editor with built-in AI-powered writing assistance tools. Using interviews and participant journals, we discuss the potential of NLG to have significant impact in the creative writing domain--especially with respect to brainstorming, generation of story details, world-building, and research assistance. Experienced writers, more so than amateurs, typically have well-developed systems and methodologies for writing, as well as distinctive voices and target audiences. Our work highlights the challenges in building for these writers; NLG technologies struggle to preserve style and authorial voice, and they lack deep understanding of story contents. In order for AI-powered writing assistants to realize their full potential, it is essential that they take into account the diverse goals and expertise of human writers.

Motivation & Objective

• Assess how professional writers interact with an AI-powered writing assistant across diverse backgrounds and genres.
• Understand workflows, including brainstorming, drafting, and world-building, when using Wordcraft.
• Identify limitations in style, voice, and content understanding that affect usefulness for experts.

Proposed method

• Commission 13 published writers from diverse backgrounds to use Wordcraft over eight weeks to write a 1,000-1,500 word story.
• Collect open-ended qualitative feedback via journals and two interviews per participant.
• Code journals and interviews to extract themes about workflows, mental models, and interface/AI limitations.
• Describe Wordcraft controls and chatbot interface and analyze how they influence writing processes.

Experimental results

Research questions

• RQ1 How do professional writers incorporate AI-assisted tools like Wordcraft into their writing workflows?
• RQ2 What workflows emerge when using AI-generated suggestions for brainstorming, world-building, and drafting?
• RQ3 What are the main limitations of AI writing assistants in preserving voice, avoiding tropes, and understanding story content?
• RQ4 How does the user interface and chatbot interaction affect expert writers’ collaboration with AI?
• RQ5 What improvements would enable AI tools to better support professional writers?

Key findings

• Writers used Wordcraft as brainstorming partner, co-writer, beta reader, and research assistant, with varying emphasis across individuals.
• Interface design and controllability of the AI significantly influence productive collaboration and adoption of the tool.
• AI generated content often lacked distinctive voice and varied in quality, with a tendency toward bland or trope-heavy suggestions.
• Participants demonstrated emergent workflows, including using AI for ideation, world-building details, rewrites, and as an improv partner, but often needed to curate and edit AI output themselves.
• The chatbot interface was valued for brainstorming but struggled to provide meaningful, opinionated guidance on story contents or stylistic choices.
• Smaller, less sophisticated models sometimes produced more interesting errors; larger models offered coherence but hindered stylistic control and novelty.