[Paper Review] Prompting Diverse Ideas: Increasing AI Idea Variance

Lennart Meincke, Ethan Mollick+1 more|arXiv (Cornell University)|Jan 27, 2024

Ethics and Social Impacts of AI4 citations

TL;DR

This paper investigates prompt engineering techniques to increase the diversity of AI-generated ideas in creative problem-solving, using GPT-4 to generate product ideas for college students under $50. It finds that Chain-of-Thought prompting significantly boosts idea dispersion and uniqueness, approaching human group performance in diversity while outperforming other prompting methods.

ABSTRACT

Unlike routine tasks where consistency is prized, in creativity and innovation the goal is to create a diverse set of ideas. This paper delves into the burgeoning interest in employing Artificial Intelligence (AI) to enhance the productivity and quality of the idea generation process. While previous studies have found that the average quality of AI ideas is quite high, prior research also has pointed to the inability of AI-based brainstorming to create sufficient dispersion of ideas, which limits novelty and the quality of the overall best idea. Our research investigates methods to increase the dispersion in AI-generated ideas. Using GPT-4, we explore the effect of different prompting methods on Cosine Similarity, the number of unique ideas, and the speed with which the idea space gets exhausted. We do this in the domain of developing a new product development for college students, priced under $50. In this context, we find that (1) pools of ideas generated by GPT-4 with various plausible prompts are less diverse than ideas generated by groups of human subjects (2) the diversity of AI generated ideas can be substantially improved using prompt engineering (3) Chain-of-Thought (CoT) prompting leads to the highest diversity of ideas of all prompts we evaluated and was able to come close to what is achieved by groups of human subjects. It also was capable of generating the highest number of unique ideas of any prompt we studied.

Motivation & Objective

• To address the limited diversity in AI-generated ideas despite high average quality, especially in creative innovation tasks.
• To investigate whether prompt engineering can increase the dispersion and uniqueness of ideas generated by large language models.
• To compare the effectiveness of different prompting strategies—especially Chain-of-Thought—in enhancing idea variance.
• To evaluate how quickly the idea space is exhausted under different prompting methods.
• To benchmark AI-generated idea diversity against human group performance in idea generation.

Proposed method

• Employed GPT-4 to generate product ideas for college students under $50 using various prompting techniques.
• Applied Chain-of-Thought (CoT) prompting, which encourages step-by-step reasoning to explore multiple idea pathways.
• Measured idea diversity using Cosine Similarity across generated idea embeddings to quantify similarity and dispersion.
• Tracked the number of unique ideas produced per prompt to assess idea space coverage and exhaustion rate.
• Used a controlled experimental design with multiple prompt variants to compare diversity outcomes.
• Benchmarked AI-generated idea pools against those from human subject groups in the same domain.

Experimental results

Research questions

• RQ1 Can prompt engineering significantly increase the diversity of ideas generated by large language models?
• RQ2 How does Chain-of-Thought prompting compare to other prompting methods in enhancing idea dispersion?
• RQ3 To what extent can AI-generated idea pools match the diversity achieved by human groups?
• RQ4 How quickly does the idea space become exhausted under different prompting strategies?
• RQ5 What is the relationship between prompting method and the number of unique ideas generated?

Key findings

• Pools of AI-generated ideas using standard prompts were significantly less diverse than those from human subject groups, as measured by Cosine Similarity.
• Chain-of-Thought prompting produced the highest diversity of ideas among all tested prompting methods.
• CoT prompting generated the highest number of unique ideas across all conditions studied.
• CoT prompting enabled AI to come close to matching the idea diversity achieved by human groups in the same task.
• Other prompting methods, including few-shot and direct prompting, resulted in lower idea dispersion and faster exhaustion of the idea space.
• The study confirms that prompt engineering can substantially improve the variance of AI-generated ideas, especially when using reasoning-based prompting.