[Paper Review] Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Games

Eilam Shapira, Omer Madmon+2 more|arXiv (Cornell University)|Jan 30, 2024

Stock Market Forecasting MethodsDecision Sciences3 citations

TL;DR

This paper demonstrates that large language models (LLMs) can generate synthetic data to train models that predict human choices in language-based persuasion games, achieving higher accuracy than models trained on real human data when sufficient LLM-generated data is available. The approach uses LLM agents with diverse personas to simulate strategic interactions, outperforming human-data baselines and even surpassing human-data models in key settings.

ABSTRACT

Human choice prediction in economic contexts is crucial for applications in marketing, finance, public policy, and more. This task, however, is often constrained by the difficulties in acquiring human choice data. With most experimental economics studies focusing on simple choice settings, the AI community has explored whether LLMs can substitute for humans in these predictions and examined more complex experimental economics settings. However, a key question remains: can LLMs generate training data for human choice prediction? We explore this in language-based persuasion games, a complex economic setting involving natural language in strategic interactions. Our experiments show that models trained on LLM-generated data can effectively predict human behavior in these games and even outperform models trained on actual human data. Beyond data generation, we investigate the dual role of LLMs as both data generators and predictors, introducing a comprehensive empirical study on the effectiveness of utilizing LLMs for data generation, human choice prediction, or both. We then utilize our choice prediction framework to analyze how strategic factors shape decision-making, showing that interaction history (rather than linguistic sentiment alone) plays a key role in predicting human decision-making in repeated interactions. Particularly, when LLMs capture history-dependent decision patterns similarly to humans, their predictive success improves substantially. Finally, we demonstrate the robustness of our findings across alternative persuasion-game settings, highlighting the broader potential of using LLM-generated data to model human decision-making.

Motivation & Objective

• To investigate whether LLM-generated data can replace human choice data in training models to predict human decisions in economic settings.
• To evaluate the feasibility and effectiveness of using LLM-based agents as synthetic participants in language-based persuasion games.
• To determine whether models trained solely on LLM-generated data can outperform models trained on real human data in predicting human behavior.
• To analyze the impact of diverse LLM personas on the quality and diversity of synthetic training data.
• To explore the conditions under which LLM-based synthetic data becomes a superior alternative to real human data for human choice prediction.

Proposed method

• The study uses a language-based persuasion game framework from Apel et al. (2022), where a sender (expert) attempts to persuade a human decision-maker (DM) to accept a hotel deal using natural language messages.
• LLM-based agents are used to simulate both the sender and receiver roles, generating synthetic interaction data without any human choice data in the training set.
• Multiple persona types (e.g., neutral, overly positive, skeptical) are used to diversify the synthetic data and improve model generalization.
• A prediction model is trained exclusively on LLM-generated data to forecast human DM decisions in response to sender messages.
• Shapley values are applied to quantify the marginal contribution of each persona type to the overall predictive performance of the model.
• The approach is evaluated by comparing prediction accuracy against models trained on real human data and linguistic baselines.

Experimental results

Research questions

• RQ1 Can a model trained solely on LLM-generated data predict human choices in a language-based persuasion game as accurately as, or more accurately than, a model trained on real human data?
• RQ2 How does the diversity of LLM personas in synthetic data generation affect the predictive performance of the resulting model?
• RQ3 In which expert strategies does the LLM-generated data approach outperform or underperform compared to human data?
• RQ4 What is the marginal contribution of each persona type to the predictive quality of the synthetic dataset?
• RQ5 Can LLM-based synthetic data reduce the need for costly and privacy-sensitive human data collection in economic choice prediction?

Key findings

• A model trained exclusively on LLM-generated data outperforms a model trained on real human choice data when the LLM-generated dataset is sufficiently large.
• The LLM-based approach achieves superior prediction accuracy even in the case of a naive expert strategy (SendBest), where the expert always sends the best possible review regardless of hotel quality.
• Using a mixture of diverse LLM personas in data generation reduces the required sample size to achieve a given accuracy level compared to using only default (neutral) personas.
• Shapley value analysis reveals that all persona types contribute nearly uniformly to the predictive power of the synthetic dataset, indicating balanced and essential contributions.
• The LLM-based synthetic data approach consistently outperforms a linguistic baseline, demonstrating that context-aware agent behavior is essential for accurate human behavior modeling.
• The method significantly reduces the number of human participants needed for data collection, increasing training efficiency and scalability.