[Paper Review] AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation

Orit Shaer, Angelora Cooper+3 more|arXiv (Cornell University)|Feb 22, 2024

Team Dynamics and Performance6 citations

TL;DR

The paper investigates integrating LLMs into group Brainwriting for both divergent idea generation and convergent evaluation, presenting a collaborative Group-AI Brainwriting framework, an LLM-based evaluation engine, and an empirical study with undergraduates to assess impact on ideation and evaluation.

ABSTRACT

The growing availability of generative AI technologies such as large language models (LLMs) has significant implications for creative work. This paper explores twofold aspects of integrating LLMs into the creative process - the divergence stage of idea generation, and the convergence stage of evaluation and selection of ideas. We devised a collaborative group-AI Brainwriting ideation framework, which incorporated an LLM as an enhancement into the group ideation process, and evaluated the idea generation process and the resulted solution space. To assess the potential of using LLMs in the idea evaluation process, we design an evaluation engine and compared it to idea ratings assigned by three expert and six novice evaluators. Our findings suggest that integrating LLM in Brainwriting could enhance both the ideation process and its outcome. We also provide evidence that LLMs can support idea evaluation. We conclude by discussing implications for HCI education and practice.

Motivation & Objective

• Explore how LLMs can augment the divergence stage of group Brainwriting to enhance idea generation and the resulting solution space.
• Investigate how LLMs can assist in the convergence stage by evaluating and selecting ideas.
• Design and evaluate a collaborative Group-AI Brainwriting framework.
• Develop an LLM-based evaluation engine that rates idea quality on predefined criteria.
• Provide empirical insights for HCI education and practice regarding human-AI co-creation.

Proposed method

• Design of a collaborative, multi-phase Group-AI Brainwriting framework integrating an LLM into divergence for idea generation and into convergence for idea enhancement.
• Use of Conceptboard as the online workspace for parallel idea generation and sharing.
• Engagement of GPT-3/GPT-4 to generate and refine ideas, with prompts refined through prompt engineering training.
• Development of an LLM evaluation engine that scores ideas on relevance, innovation, and insightfulness using a Likert scale with well-defined anchors.
• Empirical evaluation in an undergraduate tangible interaction design course with qualitative and quantitative measures, plus comparison against expert and novice human ratings.
• Semantic analysis of divergence via NLP methods (spaCy, topic modeling, LPA) and comparison of human vs. LLM-generated ideas.

Experimental results

Research questions

• RQ1 RQ1: Does the use of an LLM during the divergence stage of collaborative group Brainwriting enhance the idea generation process and its outcome?
• RQ2 RQ2: How can LLMs assist to evaluate ideas during the convergence stage of a collaborative group Brainwriting process?

Key findings

• Integrating LLMs in the divergence stage can enhance idea generation and expand the solution space, as reflected in the distribution and content of ideas.
• Participants reported that GPT-3 contributed unique viewpoints and assisted in generating new ideas beyond initial human ideas.
• An LLM-based evaluation engine (GPT-4) can rate ideas on relevance, innovation, and insightfulness with defined criteria and scale, and its outputs were compared to expert and novice human evaluators.
• The approach yields evidence that LLMs can assist in idea evaluation during convergence, aiding in identifying promising ideas for further development.
• The framework contributes to HCI education by expanding pedagogical methods and offering AI-augmented tools for educators and novice designers, while also highlighting merits and limitations of human-AI collaborative ideation.