[Paper Review] AI for the Generation and Testing of Ideas Towards an AI Supported Knowledge Development Environment

Ted Selker|arXiv (Cornell University)|Jul 17, 2023

Scientific Computing and Data Management4 citations

TL;DR

This paper proposes a hybrid AI system, Generate And Search Test (GASt), that combines generative AI's idea creation with search-based fact verification to support knowledge workers in developing reliable, traceable ideas. By integrating generative drafting with provenance-aware search, the system enhances idea generation while ensuring accuracy, reducing human bias, and enabling iterative refinement of solutions.

ABSTRACT

New systems employ Machine Learning to sift through large knowledge sources, creating flexible Large Language Models. These models discern context and predict sequential information in various communication forms. Generative AI, leveraging Transformers, generates textual or visual outputs mimicking human responses. It proposes one or multiple contextually feasible solutions for a user to contemplate. However, generative AI does not currently support traceability of ideas, a useful feature provided by search engines indicating origin of information. The narrative style of generative AI has gained positive reception. People learn from stories. Yet, early ChatGPT efforts had difficulty with truth, reference, calculations, and aspects like accurate maps. Current capabilities of referencing locations and linking to apps seem to be better catered by the link-centric search methods we've used for two decades. Deploying truly believable solutions extends beyond simulating contextual relevance as done by generative AI. Combining the creativity of generative AI with the provenance of internet sources in hybrid scenarios could enhance internet usage. Generative AI, viewed as drafts, stimulates thinking, offering alternative ideas for final versions or actions. Scenarios for information requests are considered. We discuss how generative AI can boost idea generation by eliminating human bias. We also describe how search can verify facts, logic, and context. The user evaluates these generated ideas for selection and usage. This paper introduces a system for knowledge workers, Generate And Search Test, enabling individuals to efficiently create solutions previously requiring top collaborations of experts.

Motivation & Objective

• To address the lack of idea traceability in generative AI systems, which currently obscure the origin of generated content.
• To reduce human cognitive bias in idea generation by leveraging AI to propose diverse, contextually relevant alternatives.
• To combine the creative potential of generative AI with the factual reliability of search engines for robust knowledge development.
• To design a system that supports knowledge workers in efficiently generating and testing ideas without relying on expert collaboration.
• To enable users to evaluate, refine, and validate AI-generated ideas through integrated search and provenance tracking.

Proposed method

• The system uses large language models (LLMs) to generate multiple contextually feasible ideas in response to user queries.
• Generated ideas are treated as drafts, encouraging iterative refinement and user evaluation.
• Search engines are integrated to verify factual claims, logical consistency, and contextual accuracy of generated ideas.
• The architecture supports hybrid workflows where AI generates ideas and search provides traceable sources and references.
• The system emphasizes narrative-style outputs to enhance user comprehension and engagement.
• Users can link generated ideas to external sources, ensuring provenance and enabling auditability of reasoning.

Experimental results

Research questions

• RQ1 How can generative AI be enhanced to support traceable idea development without compromising creativity?
• RQ2 In what ways can search-based verification improve the reliability of AI-generated ideas?
• RQ3 How does integrating generative AI with search reduce human bias in idea generation?
• RQ4 What role does narrative structure play in enhancing user engagement and understanding of AI-generated ideas?
• RQ5 How can a hybrid system outperform standalone generative or search-based approaches in knowledge development tasks?

Key findings

• The integration of generative AI with search enables the creation of ideas that are both creative and factually traceable.
• Users can evaluate and refine AI-generated ideas with confidence, thanks to linked sources and verifiable claims.
• The system reduces reliance on expert collaboration by automating idea generation and validation workflows.
• Narrative-style outputs from generative AI are perceived as more engaging and easier to understand than raw data or lists.
• The approach mitigates common weaknesses of early LLMs, such as factual inaccuracy and poor reasoning, through external verification.
• The hybrid model supports iterative idea development, allowing users to evolve drafts into reliable, actionable solutions.