[Paper Review] Artificial Intelligence and Strategic Decision-Making: Evidence from Entrepreneurs and Investors

Felipe A. Csaszar, Harsh Ketkar+1 more|arXiv (Cornell University)|Aug 16, 2024

Stock Market Forecasting Methods13 citations

TL;DR

The paper investigates how AI, via large language models (LLMs), can generate and evaluate strategic decisions, showing AI-plans can perform comparably to human entrepreneurs and investors in realistic settings and outlining AI’s impact on SDM processes.

ABSTRACT

This paper explores how artificial intelligence (AI) may impact the strategic decision-making (SDM) process in firms. We illustrate how AI could augment existing SDM tools and provide empirical evidence from a leading accelerator program and a startup competition that current Large Language Models (LLMs) can generate and evaluate strategies at a level comparable to entrepreneurs and investors. We then examine implications for key cognitive processes underlying SDM -- search, representation, and aggregation. Our analysis suggests AI has the potential to enhance the speed, quality, and scale of strategic analysis, while also enabling new approaches like virtual strategy simulations. However, the ultimate impact on firm performance will depend on competitive dynamics as AI capabilities progress. We propose a framework connecting AI use in SDM to firm outcomes and discuss how AI may reshape sources of competitive advantage. We conclude by considering how AI could both support and challenge core tenets of the theory-based view of strategy. Overall, our work maps out an emerging research frontier at the intersection of AI and strategy.

Motivation & Objective

• Motivate how AI could augment strategic decision-making (SDM) tools and processes.
• Provide empirical evidence on LLMs generating and evaluating entrepreneurial strategies.
• Theorize how AI affects cognitive SDM processes: search, representation, and aggregation.
• Discuss implications for competitive advantage and theory-based strategy perspectives.

Proposed method

• Reimagine four SDM tools (Scenario Planning, Porter’s Five Forces, Devil’s Advocate, Wisdom of the Crowd) with AI augmentation using GPT-4 to illustrate potential applications.
• Conduct two empirical studies to compare AI-generated versus human-generated strategies in realistic settings: a startup accelerator generation experiment (GPT-3.5) and a startup competition evaluation study (LLM versus human judges).
• Use a within-subject experimental design where evaluators assess both AI- and human-generated plans to control for evaluator effects.
• Analyze outcomes with linear specifications comparing AI versus non-AI plans and with evaluation metrics such as scores, acceptance, interest, and investment likelihood.
• Link AI-generated plan quality to real-world accelerator decisions to validate signal relevance.

Experimental results

Research questions

• RQ1 How may AI-augmented SDM look in practice using existing tools?
• RQ2 How effective are current LLMs at generating and evaluating strategic plans in realistic entrepreneurial settings?
• RQ3 What are the implications of using AI in SDM for cognitive processes and competitive outcomes?
• RQ4 How do AI-generated strategies compare to human-generated strategies in terms of investor interest and decision outcomes?

Key findings

• LLM-generated business plans were rated higher by 0.14 standard deviations on average across key attributes (p<0.001).
• Evaluators were 5 percentage points more likely to recommend accepting LLM-generated plans (p=0.003).
• LLM-generated plans were 3 percentage points more likely to result in investor introductions (p=0.094) and 3 percentage points more likely to attract investment (p=0.007).
• In a separate accelerator analysis, LLM-generated plans outperformed rejected entrepreneur plans by 7–8 percentage points on acceptance and interest metrics (p=0.003 and p=0.001) and by 6 percentage points on investment (p<0.001).
• LLM evaluations of 138 business plans in a startup competition correlated with human investor scores (average correlation 0.52; ICC 0.51 between AI and VC scores, higher than inter-rater reliability among human judges).
• Across indicators, AI scores explain about 29% of the variation in investor scores for the evaluated plans.