[Paper Review] Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking

Harsh Kumar, Jonathan Vincentius+2 more|arXiv (Cornell University)|Sep 24, 2024

Private Equity and Venture CapitalBusiness, Management and Accounting3 citations

TL;DR

This study investigates how large language models (LLMs) affect human creativity through two pre-registered randomized experiments with 1,100 participants, comparing standard LLM assistance, coach-like LLM guidance, and no assistance. It finds that while LLMs boost creativity during assisted tasks, they may impair independent creative performance in unassisted settings, suggesting potential long-term cognitive dependency on AI tools.

ABSTRACT

Large language models are transforming the creative process by offering unprecedented capabilities to algorithmically generate ideas. While these tools can enhance human creativity when people co-create with them, it's unclear how this will impact unassisted human creativity. We conducted two large pre-registered parallel experiments involving 1,100 participants attempting tasks targeting the two core components of creativity, divergent and convergent thinking. We compare the effects of two forms of large language model (LLM) assistance -- a standard LLM providing direct answers and a coach-like LLM offering guidance -- with a control group receiving no AI assistance, and focus particularly on how all groups perform in a final, unassisted stage. Our findings reveal that while LLM assistance can provide short-term boosts in creativity during assisted tasks, it may inadvertently hinder independent creative performance when users work without assistance, raising concerns about the long-term impact on human creativity and cognition.

Motivation & Objective

• To investigate how different forms of LLM assistance affect human creativity in controlled, experimental settings.
• To assess whether LLMs enhance or impair unassisted creative performance after prior exposure to AI assistance.
• To compare the impact of direct LLM answers versus coach-like guidance on divergent and convergent thinking.
• To evaluate the long-term cognitive effects of AI use on human creative potential.
• To provide empirical evidence on whether AI assistance fosters or undermines independent human creativity.

Proposed method

• Conducted two pre-registered, parallel experiments with 1,100 participants, randomly assigned to three conditions: no assistance, standard LLM assistance, or coach-like LLM guidance.
• Used the Alternate Uses Test (AUT) to measure divergent thinking and the Remote Associates Test (RAT) to measure convergent thinking.
• Participants completed a series of exposure rounds with assigned LLM assistance, followed by a final unassisted test round to assess residual creative performance.
• Employed a within-subjects design with a delay period between exposure and test phases to isolate long-term effects.
• Collected and analyzed qualitative and quantitative outputs for creativity metrics, including idea quantity, originality, and solution quality.
• Controlled for verbosity and guidance style in the coach-like LLM to isolate the impact of interaction design on cognitive outcomes.

Figure 1. Experimental framework for measuring the impact of AI use on Human creativity. Participants engage in a series of Exposure rounds where they are randomized to either receive - (A) No assistance, (B) LLM solution (standard): This could be analogous to using a chat LLM such as ChatGPT for th

Experimental results

Research questions

• RQ1 RQ1: How do standard LLM assistance and coach-like LLM guidance, compared to no assistance, affect an individual’s divergent thinking abilities when generating creative ideas independently?
• RQ2 RQ2: What are the impacts of standard LLM assistance and coach-like LLM guidance, versus no assistance, on an individual’s convergent thinking skills in independently refining and selecting ideas?
• RQ3 RQ3: How does prior exposure to LLM assistance influence unassisted creative performance in subsequent tasks?
• RQ4 RQ4: Does the mode of LLM interaction (direct answer vs. guided thinking) differentially affect long-term human creative cognition?

Key findings

• Participants receiving standard LLM assistance showed higher idea generation during assisted tasks but performed worse in the final unassisted test round compared to the control group.
• Coach-like LLM guidance led to better performance in the unassisted test phase than direct LLM assistance, suggesting that prompting for reflection may preserve independent creativity.
• The control group outperformed both LLM-assisted groups in the final unassisted round, indicating that AI assistance may impair independent creative performance over time.
• The study observed a significant decline in originality and idea quantity in unassisted tasks following exposure to LLMs, especially with direct answer models.
• Participants using coach-like LLMs generated more conceptually distinct ideas in the test phase, implying that process-oriented assistance may better support cognitive retention of creative skills.
• The results suggest that over-reliance on LLMs for direct answers may lead to cognitive atrophy in independent creative thinking, even after brief exposure.

Figure 2. Schematic of design for Experiment 1 on divergent thinking.