[Paper Review] SSFF: Investigating LLM Predictive Capabilities for Startup Success through a Multi-Agent Framework with Enhanced Explainability and Performance

Xisen Wang, Yigit Ihlamur+1 more|arXiv (Cornell University)|May 29, 2024

Private Equity and Venture CapitalBusiness, Management and Accounting3 citations

TL;DR

This paper introduces the Startup Success Forecasting Framework (SSFF), an LLM-powered multi-agent system that autonomously evaluates early-stage startups by integrating external knowledge, predictive machine learning (random forests, neural networks), and advanced prompting. It achieves high-accuracy, explainable forecasts comparable to expert VCs using minimal input data and real-time market intelligence.

ABSTRACT

LLM based agents have recently demonstrated strong potential in automating complex tasks, yet accurately predicting startup success remains an open challenge with few benchmarks and tailored frameworks. To address these limitations, we propose the Startup Success Forecasting Framework, an autonomous system that emulates the reasoning of venture capital analysts through a multi agent collaboration model. Our framework integrates traditional machine learning methods such as random forests and neural networks within a retrieval augmented generation framework composed of three interconnected modules: a prediction block, an analysis block, and an external knowledge block. We evaluate our framework and identify three main findings. First, by leveraging founder segmentation, startups led by L5 founders are 3.79 times more likely to succeed than those led by L1 founders. Second, baseline large language models consistently overpredict startup success and struggle under realistic class imbalances largely due to overreliance on founder claims. Third, our framework significantly enhances prediction accuracy, yielding a 108.3 percent relative improvement over GPT 4o mini and a 30.8 percent relative improvement over GPT 4o. These results demonstrate the value of a multi agent approach combined with discriminative machine learning in mitigating the limitations of standard large language model based prediction methods.

Motivation & Objective

• To address the challenge of automating early-stage startup evaluation, traditionally reliant on expert intuition and prone to bias.
• To overcome limitations of LLMs in predictive reliability—such as hallucination and fuzzy semantics—by combining them with structured machine learning.
• To develop a scalable, automated pipeline that enhances forecasting accuracy using minimal founder input and real-time external data.
• To create a modular, explainable agent architecture that mimics VC decision-making processes end-to-end.
• To establish a benchmark for AI-driven venture capital analysis by integrating qualitative and quantitative assessment in a unified framework.

Proposed method

• The SSFF employs a three-block architecture: Prediction Block (random forests and neural networks), Analyst Block (LLM agents using SOTA prompting for VC-style reasoning), and External Knowledge Block (real-time data retrieval).
• A VC scout agent first synthesizes startup input into 18 critical dimensions, enabling structured analysis across multiple analytical agents.
• The framework uses a trained decision tree model to generate preliminary success predictions based on 14 tailored questions.
• Founder-level segmentation and clustering are applied to refine founder-specific insights, while a Founder-Idea Fit model computes a quantitative fit score via embeddings.
• External Knowledge Block retrieves current market intelligence and news, enriching data for both market and product analysts.
• A chief analyst agent integrates all outputs into a comprehensive, explainable report synthesizing predictive analytics and strategic insights.

Experimental results

Research questions

• RQ1 Can a multi-agent LLM framework effectively simulate venture capitalist reasoning in evaluating early-stage startups with minimal input?
• RQ2 How does combining traditional machine learning (random forests, neural networks) with LLMs improve predictive accuracy compared to zero-shot LLMs?
• RQ3 To what extent does real-time external knowledge retrieval enhance the robustness and relevance of startup success forecasts?
• RQ4 Can explainable AI components (e.g., decision trees, clustering, embedding-based fit scores) improve transparency and trust in automated startup evaluation?
• RQ5 How does the modular, agent-based design of SSFF support scalability, adaptability, and future integration of new data sources?

Key findings

• The SSFF significantly outperforms zero-shot GPT responses in efficiency, thoroughness, depth of analysis, reliability, and timeliness.
• The framework achieves predictive accuracy comparable to or exceeding that of experienced venture capital analysts across diverse startup cases.
• The integration of real-time external knowledge and market intelligence enhances forecast relevance and reflects dynamic market conditions.
• The Founder-Idea Fit model, based on embeddings and refinement techniques, successfully captures unique success patterns tied to founder-background and idea alignment.
• The Fuzzy Random Forest algorithm demonstrates novel effectiveness in handling uncertainty and imprecise inputs common in startup data.
• The modular architecture enables future extensibility, including integration with data rooms, CRM systems, and product logs for deeper due diligence.