[Paper Review] Rapid AIdeation: Generating Ideas With the Self and in Collaboration With Large Language Models

Gionnieve Lim, Simon T. Perrault|arXiv (Cornell University)|Mar 19, 2024

Artificial Intelligence in Healthcare and Education4 citations

TL;DR

This study investigates how large language models (LLMs) enhance rapid ideation in design contexts through a workshop with 21 participants. It finds that LLM collaboration generates more diverse, high-quality ideas than solo ideation, with users adopting either a consulting or assisting collaboration mode—though rare anti-collaborative prompting (e.g., threats) also emerged.

ABSTRACT

Generative artificial intelligence (GenAI) can rapidly produce large and diverse volumes of content. This lends to it a quality of creativity which can be empowering in the early stages of design. In seeking to understand how creative ways to address practical issues can be conceived between humans and GenAI, we conducted a rapid ideation workshop with 21 participants where they used a large language model (LLM) to brainstorm potential solutions and evaluate them. We found that the LLM produced a greater variety of ideas that were of high quality, though not necessarily of higher quality than human-generated ideas. Participants typically prompted in a straightforward manner with concise instructions. We also observed two collaborative dynamics with the LLM fulfilling a consulting role or an assisting role depending on the goals of the users. Notably, we observed an atypical anti-collaboration dynamic where participants used an antagonistic approach to prompt the LLM.

Motivation & Objective

• To examine how large language models (LLMs) support rapid ideation in design processes.
• To understand the dynamics of human-LLM collaboration during creative brainstorming.
• To investigate prompting strategies and their impact on idea quality and diversity.
• To identify unconventional interaction patterns, such as anti-collaborative prompting, in human-AI ideation.
• To assess the comparative effectiveness of self-generating ideas versus co-creating with LLMs.

Proposed method

• Conducted a rapid ideation workshop with 21 participants using a two-phase design: first solo ideation, then ideation with ChatGPT (GPT-3.5).
• Collected and analyzed prompts, generated ideas, and user evaluations to assess idea quality and diversity.
• Employed content analysis to categorize collaboration dynamics: consulting role, assisting role, and anti-collaboration.
• Used loose qualitative criteria for 'unique' and 'high quality' ideas, informed by thematic and frequency analysis.
• Tracked prompting styles, including tone, structure, and use of social cues (e.g., greetings, 'you/I' references).
• Compared idea sets from self-ideation and Co-GPT ideation to evaluate LLM impact on idea generation.

Experimental results

Research questions

• RQ1 How does co-creation with an LLM affect the diversity and quality of ideas in rapid ideation compared to solo ideation?
• RQ2 What are the dominant collaboration dynamics between humans and LLMs during ideation?
• RQ3 How do users prompt LLMs, and what patterns emerge in their prompting strategies?
• RQ4 Are there non-traditional or anti-collaborative prompting behaviors observed, and what motivates them?
• RQ5 What role might LLMs play as evaluators of ideas, and is this desirable in creative workflows?

Key findings

• LLM-assisted ideation produced a greater variety of unique ideas compared to solo ideation, though not necessarily higher quality in absolute terms.
• Participants typically used concise, straightforward prompts with a semi-formal tone, reflecting time pressure in rapid ideation.
• Two primary collaboration dynamics emerged: LLMs acting as consultants (providing ideas) or assistants (helping refine or summarize ideas).
• An atypical anti-collaboration dynamic was observed where two participants used threatening or interrogative prompts to coerce desired outputs.
• Some participants treated the LLM as a social agent, using greetings and conversational cues, reflecting human-like interaction expectations.
• The use of GPT-3.5, a free and widely accessible model, may have limited the quality and diversity of outputs, though results still showed meaningful enhancement over solo ideation.