[Paper Review] Human-LLM Compound System for Scientific Ideation through Facet Recombination and Novelty Evaluation

Marissa Radensky, Simra Shahid+4 more|arXiv (Cornell University)|Sep 23, 2024

Scientific Computing and Data Management5 citations

TL;DR

Scideator is a mixed-initiative tool that extracts facets from input and analogous papers to recombine into potentially novel scientific ideas, with an LLM-powered novelty evaluation to guide iteration. In a user study, it yielded more novel and interesting ideas than a strong baseline.

ABSTRACT

The scientific ideation process often involves blending salient aspects of existing papers to create new ideas - a framework known as facet-based ideation. We contribute Scideator, the first human-LLM system for facet-based scientific ideation. Starting from a user-provided set of scientific papers, Scideator extracts key facets -- purposes, mechanisms, and evaluations -- from these and related papers, allowing users to explore the idea space by interactively recombining facets to synthesize inventive ideas. Scideator is driven by three design choices: (1) human-in-the-loop facet recombination, in which users select facets from retrieved papers and the system generates ideas by finding analogies across them via the Faceted Idea Generator module; (2) distance-controlled retrieval via the Analogous Paper Facet Finder module, which surfaces papers from the same topic to entirely different subareas to provide a spectrum of creative directions; and (3) facet-based novelty verification via the Idea Novelty Checker module, a retrieve-then-rerank pipeline that evaluates idea originality using facets. In a user study with computer science researchers, Scideator provided significantly more creativity support than a baseline using the same backbone LLM without our facet-based modules, particularly in idea exploration and expressiveness. Participants' favorite ideas more often included facets selected by themselves rather than the LLM, and participants used fewer free-text instructions with Scideator, indicating a preference for facet-level steering over prompting. Finally, re-ranking papers by facet matching rather than general relevance improved novelty classification accuracy from 13.79% to 89.66%.

Motivation & Objective

• Motivate and enable scientists to generate relevant and novel research ideas grounded in literature.
• Extract and reuse facets (purpose, mechanism, evaluation) from input and analogous papers to guide idea generation.
• Provide automated novelty assessment to enable iterative improvement of ideas.
• Evaluate the impact of Scideator on researchers' ideation quality through a within-subjects study.
• Offer insights into design choices and user interaction patterns in human-LLM scientific ideation.

Proposed method

• Introduce four LLM-powered retrieval-augmented generation modules: Analogous Paper Facet Finder, Faceted Idea Generator, Idea Novelty Checker, and Idea Novelty Iterator.
• Extract facets (purpose, mechanism, evaluation) from input and analogous papers using Semantic Scholar data and LLM reasoning.
• Generate ideas by recombining selected facets and distances (near/far analogies) across input and analogous papers.
• Assess novelty by retrieving related papers and producing a reasoned novelty classification with user-facing explanations.
• Provide iterative novelty improvements by suggesting more novel ideas when an idea is deemed not novel.
• Conduct a within-subjects user study (N=19 CS researchers) comparing Scideator against a strong literature-search plus LLM baseline.
• Analyze interaction patterns, facet distance usage, and novelty evaluation performance, including comparisons to prior novelty methods.

Figure 1 . The Scideator workflow. 1) The interaction starts with the user providing a set of input papers that they would like to use as a starting point for ideation. 2) The tool responds by retrieving analogous papers to the input papers and extracting facets from the input and analogous papers.

Experimental results

Research questions

• RQ1 How does Scideator support the generation of novel and relevant scientific ideas grounded in the literature?
• RQ2 Does the facet-recombination approach plus automated novelty evaluation lead to more novel ideas than a baseline approach?
• RQ3 How accurate and helpful is the automated novelty checker relative to human judgments and prior methods?
• RQ4 What are user interaction patterns and design factors that influence ideation outcomes (e.g., facet distance, saving behavior)?
• RQ5 What qualitative and quantitative effects do design choices (e.g., manual novelty annotation) have on novelty assessment and idea generation?

Key findings

• Participants generated more novel and interesting ideas with Scideator than with the baseline (after removing an outlier, result is statistically significant).
• The Idea Novelty Checker improved agreement with human novelty ratings by more than 10x compared to a recent LLM-based novelty approach and about 13% higher agreement than another recent method.
• In the within-subjects study, N=19 CS researchers produced more ideas rated as novel and interesting when using Scideator versus the baseline.
• Manual annotation of novelty and concise reviews yielded better novelty evaluation outcomes than baselines relying solely on literature retrieval or prompt-optimizer approaches.
• Analysis of interaction logs revealed that participants spent more time with saved ideas and facets, with junior researchers engaging more with distant facets and unfamiliar ideas, and seniors showing different patterns of exploration.
• The system design supports two goals: relevance (through facet retrieval and user selection) and novelty (through facet distance and automated novelty reasoning).

(a)