[论文解读] Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas

Xiang Hu, Hongyu Fu+8 more|arXiv (Cornell University)|Oct 18, 2024

Biomedical and Engineering Education被引用 5

一句话总结

Nova introduces an iterative planning and external-knowledge retrieval framework to boost novelty and diversity in LLM-generated research ideas, achieving 3.4x more unique novel ideas and at least 2.5x more top-rated ideas than baselines.

ABSTRACT

Scientific innovation is pivotal for humanity, and harnessing large language models (LLMs) to generate research ideas could transform discovery. However, existing LLMs often produce simplistic and repetitive suggestions due to their limited ability in acquiring external knowledge for innovation. To address this problem, we introduce an enhanced planning and search methodology designed to boost the creative potential of LLM-based systems. Our approach involves an iterative process to purposely plan the retrieval of external knowledge, progressively enriching the idea generation with broader and deeper insights. Validation through automated and human assessments indicates that our framework substantially elevates the quality of generated ideas, particularly in novelty and diversity. The number of unique novel ideas produced by our framework is 3.4 times higher than without it. Moreover, our method outperforms the current state-of-the-art, generating at least 2.5 times more top-rated ideas based on 170 seed papers in a Swiss Tournament evaluation.

研究动机与目标

• 激发并帮助 LLM 生成更高质量、更新颖的科学研究创意。
• 通过设计目标导向的外部知识检索来解决重复性想法。
• 开发一个迭代规划流程，规划知识检索以丰富创意生成。
• 通过自动和人工评估证明基于规划的知识检索的有效性。）

提出的方法

• 一个迭代的规划与检索框架，促使 LLM 设计面向新颖性和多样性的知识检索计划。
• 使用相关文献和科学发现方法进行多源种子想法生成以产生初始想法（每个输入论文 15 个种子想法）。
• 迭代循环中，每次迭代计划检索、获取外部论文、生成新种子并使用自我反思来裁剪想法（每次迭代 3 条）。
• 最终阶段通过将想法分解为子模块并详细描述方法学，将种子想法扩展为初步和最终提案。
• 使用瑞士制对战式排名（Claude-3.5-Sonnet 零-shot 排名器）进行质量的自动评估，以及基于主题相似论文和余弦相似度阈值的新颖性/多样性指标。
• 让 NLP/ML/CV 专家进行的人类评估，评估新颖性和整体质量以验证自动指标。）

Figure 1: Nova’s Performance. The Left: Comparison with the state-of-the-arts. Nova significantly outperforms other agents (Si et al., 2024 ; Baek et al., 2024 ; Lu et al., 2024 ) in generating high-quality ideas (Swiss Tournament Score is 5). The Right: The number of unique novel ideas at each iter

实验结果

研究问题

• RQ1 相较于先前的 state-of-the-art 方法，迭代式规划与外部知识检索是否提高了 LLM 生成想法的新颖性和多样性？
• RQ2 自动评估（瑞士制、新颖性、多样性指标）与人类评估在判断创意质量方面如何对齐？
• RQ3 规划组件与检索组件对观测到的收益有何贡献？
• RQ4 该框架是否能够在大规模种子论文集合上扩展并仍然产出高质量想法？

主要发现

• Nova 相比缺少迭代规划框架的基线，产生 3.4x 的更多独特新颖想法。
• Nova 在对 170 篇种子论文的瑞士锦标赛评估中，与你当前的 state-of-the-art 方法相比，至少获得 2.5x 的高评分想法。
• 自动评估显示 Nova 获得更高的 Swiss 分数以及更高比例被评为 4 或 5 的想法。
• 人类评估确认在整体质量与新颖性方面，Nova 在比较的代理中获得最高分。
• 消融研究表明规划与检索都很关键；移除规划会阻止独特想法生成的进展，而两者均移除则收益受限。

Figure 2: Nova Pipeline. The Pipeline includes initial seed idea generation, seed idea iteration, and idea completion. Upon receiving an input paper ( i.e. , seed paper), the LLM is prompted to generate initial seed ideas by utilizing related papers (including recent publications) and scientific dis