[论文解读] CREATE: Testing LLMs for Associative Creativity

Manya Wadhwa, Tania Roy+3 more|arXiv (Cornell University)|Mar 10, 2026

Artificial Intelligence in Games被引用 0

一句话总结

CREATE 通过在知识图中生成并对现实世界概念之间的高质量、多样化、独特路径进行生成与排序来评估LLMs的联想创造力； frontier 模型表现最好，但达到高度独特性仍具挑战。

ABSTRACT

A key component of creativity is associative reasoning: the ability to draw novel yet meaningful connections between concepts. We introduce CREATE, a benchmark designed to evaluate models' capacity for creative associative reasoning. CREATE requires models to generate sets of paths connecting concepts in a model's parametric knowledge. Paths should have high specificity (distinctiveness and closeness of the concept connection) and high diversity (dissimilarity from other paths), and models are scored more highly if they produce a larger set of strong, diverse paths. This task shares demands of real creativity tasks like hypothesis generation, including an extremely large search space, but enables collection of a sizable benchmark with objective answer grading. Evaluation of frontier models shows that the strongest models achieve higher creative utility than others, with the high multiplicity of answers and complexity of the search making benchmark saturation difficult to achieve. Furthermore, our results illustrate that thinking models are not always more effective on our task, even with high token budgets. Recent approaches for creative prompting give some but limited additional improvement. CREATE provides a sandbox for developing new methods to improve models' capacity for associative creativity.

研究动机与目标

• 评估LLMs在现实世界概念之间生成创意、开放式连接的能力。
• 在知识图路径中通过质量、多样性和独特性定义并衡量联想创造力。
• 研究模型思维预算与提示策略如何影响创意输出。
• 提供一个可扩展、以知识为基础的基准，以客观评分引导创意AI的发展。

提出的方法

• 将联想创造力形式化为通过有效三元组连接实体的知识图路径。
• 将质量定义为路径三元组中的最小特异性并确保关系的事实性。
• 通过路径字符串的嵌入向量余弦距离来定义路径间的距离。
• 将质量与距离结合成带有耐心参数的创意效用度量。
• 使用 Wikidata 派生查询构建覆盖多领域的 CREATE，并通过人工-LLM 判定进行验证。
• 评估一组广泛的模型（非思考与思考）并使用基础提示及变体，包括迭代提示和重采样提示。

Figure 1 : Motivating example of brainstorming paths in knowledge graphs. In CREATE, only the question is given; reasoning over the graph is implicit in the model’s parameters and thinking trace, similar to drawing connections for scientific research. Finding strong, distinct paths can be challengin

实验结果

研究问题

• RQ1 LLMs 是否能够生成多个高质量、多样且独特的连接现实世界实体的路径？
• RQ2 模型思维预算和提示变体如何影响创意效用、质量、多样性和独特性？
• RQ3 在事实性与创意效用之间的权衡是什么，哪些模型在严格条件下权衡最好？
• RQ4 先进的提示策略是否能在不同模型中稳定提升联想创造力？

主要发现

Model	s0.7	s	sigma	d	|U|	avg num tokens
GPT-4.1-mini	6.15 (5.08)	7.16 (6.81)	3.09 (1.66)	0.81 (0.26)	3.59 (3.72)	797 (258)
GPT-4.1	7.49 (5.25)	9.39 (8.01)	3.31 (1.50)	0.77 (0.27)	6.05 (5.27)	1076 (430)
GPT-5-mini (low)	6.21 (4.19)	7.03 (5.40)	3.23 (1.47)	0.64 (0.31)	4.95 (3.75)	1918 (482)
GPT-5-mini (med)	7.09 (4.61)	8.54 (6.56)	3.36 (1.45)	0.61 (0.31)	7.94 (5.52)	6360 (1743)
GPT-5-mini (high)	7.83 (4.95)	10.16 (7.85)	3.41 (1.46)	0.57 (0.29)	15.48 (10.65)	23480 (5518)
GPT-5 (med)	8.98 (5.11)	12.03 (8.67)	3.63 (1.34)	0.58 (0.27)	18.84 (13.72)	19090 (4767)
Claude-3-Haiku	3.49 (3.38)	3.68 (3.83)	2.34 (1.57)	0.83 (0.29)	1.69 (2.02)	373 (108)
Claude-Haiku-4.5 (low)	4.50 (3.78)	4.91 (4.54)	2.65 (1.51)	0.74 (0.32)	2.78 (2.79)	1004 (259)
Claude-Haiku-4.5 (med)	4.84 (3.87)	5.30 (4.67)	2.77 (1.53)	0.71 (0.31)	3.12 (3.01)	1658 (477)
Claude-Haiku-4.5 (high)	4.86 (3.97)	5.36 (4.89)	2.81 (1.55)	0.69 (0.33)	3.16 (3.03)	2150 (529)
Qwen3-30B-Instruct	5.20 (4.60)	6.27 (6.42)	2.66 (1.58)	0.75 (0.30)	5.61 (7.12)	1905 (480)
Qwen3-32B (16k)	4.69 (3.88)	5.08 (4.64)	2. unknown	0.81 (0.27)	2.34 (2.40)	3347 (1255)
Qwen3-32B (32k)	4.71 (3.77)	5.11 (4.56)	2.78 (1.51)	0.83 (0.26)	2.38 (2.43)	3333 (1221)
Olmo-3.1-32B-Instruct	3.77 (3.58)	4.13 (4.34)	2.32 (1.56)	0.83 (0.26)	2.46 (3.06)	846 (313)
Olmo-3.1-32B-Think (16k)	4.78 (3.96)	5.25 (4.95)	2.86 (1.63)	0.72 (0.33)	3.19 (3.46)	11939 (2269)
Olmo-3.1-32B-Think (32k)	4.97 (4.24)	5.52 (5.35)	2.87 (1.66)	0.71 (0.33)	3.34 (3.66)	12139 (2481)
Gemini-3-pro	8.29 (5.19)	10.41 (7.95)	3.56 (1.42)	0.77 (0.25)	6.00 (4.93)	1770 (795)

• Frontier 模型在所有耐心设置中相较于开源及较小模型，获得最高的创意效用。
• 增加生成路径数量（更高耐心）通常会提高效用，但并非在所有模型上都普遍成立。
• 更高的质量与更大程度的路径多样性与更高的效用正相关；某些模型在路径强且独特时，生成较少的路径也可达到类似效用。
• 迭代提示与重采样显著提升创意效用，而口头化采样会降低路径的有效性。
• 独特性 nu(U) 在前沿模型之间相似，但迭代比重采样更可靠地提升独特性。
• 事实性与效用存在权衡；更严格的事实性要求会降低效用，在严格条件下 GPT-5 能在两者之间取得最好平衡。
• LLM 判定的事实性判断显示出合理的可靠性，但在各类别之间 Precision/Recall 存在差异。

Figure 2 : Examples of model-generated paths $u$ compared against population paths, along with quality scores and minimum distance values. The first and last connect artists through classic relations of directing, acting, performing, etc. The second path is the weakest according to the assessed spec