[Paper Review] Using Artificial Intelligence to Accelerate Collective Intelligence: Policy Synth and Smarter Crowdsourcing

Róbert Bjarnason, Dane Gambrell+1 more|arXiv (Cornell University)|Jun 3, 2024

Big Data and Business IntelligenceBusiness, Management and Accounting3 citations

TL;DR

This paper introduces Policy Synth, an AI-powered toolkit that enhances Smarter Crowdsourcing—a method for accelerating collective intelligence in public problem-solving. By integrating AI to synthesize expert input and research, the approach boosts solution quality, scalability, and efficiency, as demonstrated in a real-world case study showing significant improvements over expert crowdsourcing alone.

ABSTRACT

In an era characterized by rapid societal changes and complex challenges, institutions' traditional methods of problem-solving in the public sector are increasingly proving inadequate. In this study, we present an innovative and effective model for how institutions can use artificial intelligence to enable groups of people to generate effective solutions to urgent problems more efficiently. We describe a proven collective intelligence method, called Smarter Crowdsourcing, which is designed to channel the collective intelligence of those with expertise about a problem into actionable solutions through crowdsourcing. Then we introduce Policy Synth, an innovative toolkit which leverages AI to make the Smarter Crowdsourcing problem-solving approach both more scalable, more effective and more efficient. Policy Synth is crafted using a human-centric approach, recognizing that AI is a tool to enhance human intelligence and creativity, not replace it. Based on a real-world case study comparing the results of expert crowdsourcing alone with expert sourcing supported by Policy Synth AI agents, we conclude that Smarter Crowdsourcing with Policy Synth presents an effective model for integrating the collective wisdom of human experts and the computational power of AI to enhance and scale up public problem-solving processes. While many existing approaches view AI as a tool to make crowdsourcing and deliberative processes better and more efficient, Policy Synth goes a step further, recognizing that AI can also be used to synthesize the findings from engagements together with research to develop evidence-based solutions and policies. The study offers practical tools and insights for institutions looking to engage communities effectively in addressing urgent societal challenges.

Motivation & Objective

• To address the limitations of traditional public sector problem-solving methods in responding to complex, urgent societal challenges.
• To develop a scalable and effective framework that integrates human expertise with AI to accelerate solution generation.
• To design a human-centric AI toolkit that enhances, rather than replaces, human intelligence in collective decision-making.
• To evaluate the impact of AI augmentation on the quality, speed, and scalability of expert-driven crowdsourcing processes.
• To provide practical tools and insights for institutions seeking to implement AI-augmented collective intelligence in policy development.

Proposed method

• Policy Synth employs AI agents to automate and enhance key stages of the Smarter Crowdsourcing process, including problem framing, expert engagement, and solution synthesis.
• The toolkit uses natural language processing and retrieval-augmented generation to integrate findings from expert inputs with external research and evidence.
• AI agents are designed to guide and structure expert contributions, ensuring coherence, relevance, and alignment with policy goals.
• The system supports iterative refinement of solutions by identifying gaps, detecting inconsistencies, and suggesting evidence-based improvements.
• A human-in-the-loop design ensures that AI augments, rather than replaces, expert judgment and creativity.
• The method is grounded in a real-world case study comparing outcomes from expert crowdsourcing alone versus expert crowdsourcing enhanced by Policy Synth.

Experimental results

Research questions

• RQ1 How can AI be effectively integrated into collective intelligence processes to improve the speed and quality of public problem-solving?
• RQ2 To what extent does AI augmentation enhance the scalability and efficiency of expert crowdsourcing in policy development?
• RQ3 Can AI agents synthesize expert input and research to generate evidence-based policy solutions more effectively than human-only processes?
• RQ4 How does the human-in-the-loop design of Policy Synth preserve expert agency while amplifying collective intelligence?
• RQ5 What measurable improvements does Policy Synth deliver in solution quality, time-to-solution, and stakeholder engagement compared to traditional methods?

Key findings

• Policy Synth significantly improved solution quality by synthesizing expert input with relevant research, resulting in more evidence-based and actionable policy recommendations.
• The AI-augmented process reduced time-to-solution compared to expert crowdsourcing alone, demonstrating enhanced efficiency in collective problem-solving.
• The toolkit increased scalability by enabling broader expert participation through structured, AI-guided contribution workflows.
• The case study showed that AI agents effectively identified gaps and inconsistencies in expert submissions, leading to more coherent and comprehensive solutions.
• Participants reported higher confidence in the final outputs when AI was used to synthesize and validate contributions, indicating improved trust and usability.
• The human-centric design of Policy Synth preserved expert control while amplifying collective intelligence, avoiding over-reliance on automation.