[Paper Review] GPT in Game Theory Experiments

Fulin Guo|arXiv (Cornell University)|May 9, 2023

Evolutionary Game Theory and Cooperation10 citations

TL;DR

The paper uses GPT-4 prompts to play finitely repeated ultimatum and prisoner’s dilemma games, analyzing how GPT traits like fairness or selfishness influence decisions and the reasoning behind them.

ABSTRACT

This paper explores the use of Generative Pre-trained Transformers (GPT) in strategic game experiments, specifically the ultimatum game and the prisoner's dilemma. I designed prompts and architectures to enable GPT to understand the game rules and to generate both its choices and the reasoning behind decisions. The key findings show that GPT exhibits behaviours similar to human responses, such as making positive offers and rejecting unfair ones in the ultimatum game, along with conditional cooperation in the prisoner's dilemma. The study explores how prompting GPT with traits of fairness concern or selfishness influences its decisions. Notably, the "fair" GPT in the ultimatum game tends to make higher offers and reject offers more frequently compared to the "selfish" GPT. In the prisoner's dilemma, high cooperation rates are maintained only when both GPT players are "fair". The reasoning statements GPT produces during gameplay reveal the underlying logic of certain intriguing patterns observed in the games. Overall, this research shows the potential of GPT as a valuable tool in social science research, especially in experimental studies and social simulations.

Motivation & Objective

• Demonstrate the feasibility of using GPT-4 in strategic game experiments.
• Explore how prompting GPT with fairness vs. selfishness affects decisions and reasoning in the ultimatum game and the prisoner’s dilemma.
• Analyze how GPT-generated reasoning explains observed behavioral patterns.
• Compare GPT-driven results to human subjects’ behavior in classic game theory experiments.

Proposed method

• Use gpt-4-1106-preview to run multi-round games with two GPT players per session.
• Assign two traits via prompts: fairness concern and selfishness, creating four ultimatum-game treatments and three prisoner’s-dilemma treatments.
• Require GPTs to output both decision and reasoning, enabling text-analysis of decision motives.
• Repeat each treatment across 100 simulations, five rounds per game, with a 1.0 temperature to promote variety.
• Analyze reasoning texts to categorize motifs like reputation building and altruism with deterministic analysis (temperature 0 for reasoning).
• Employ OLS and logit regressions to link offers, rejections, and round progression to GPT traits and round number.

Experimental results

Research questions

• RQ1 Do GPT players exhibit human-like behavior in the ultimatum game and the prisoner’s dilemma under finitely repeated play?
• RQ2 How do fairness vs. selfishness prompts influence GPT decisions and rejection/cooperation patterns?
• RQ3 What explanations can be derived from GPT-generated reasoning about observed patterns (e.g., non-monotonic rejection in round 3, high cooperation when both are fair)?

Key findings

• In the ultimatum game, fair GPTs offer around 40% while selfish GPTs offer around 30% on average.
• Rejection rates are higher for fair responders, with the SF combination showing the highest rejection (~18%).
• Proposers adjust offers downward if the previous round was accepted and tend to increase offers after a rejection.
• In the prisoner’s dilemma, cooperation rates are near 100% only when both players are fair; a single selfish GPT disrupts cooperation significantly.
• Cooperation is largely driven by reputation-building cues rather than altruism, especially in early rounds.
• Reasoning texts reveal strategic considerations like anticipating future offers and endgame dynamics, aiding interpretation of observed patterns.