[Paper Review] Online Learning in Iterated Prisoner's Dilemma to Mimic Human Behavior

Baihan Lin, Djallel Bouneffouf+1 more|arXiv (Cornell University)|Jun 9, 2020

Evolutionary Game Theory and Cooperation4 citations

TL;DR

This paper evaluates online learning algorithms—multi-armed bandits, contextual bandits, and reinforcement learning—in the Iterated Prisoner’s Dilemma (IPD) to assess their performance in social dilemmas and their ability to mimic human behavior. It finds that contextual bandits underperform due to over-reliance on current context, while simple bandits best fit human decision-making, suggesting humans may not use situational context in IPD.

ABSTRACT

As an important psychological and social experiment, the Iterated Prisoner's Dilemma (IPD) treats the choice to cooperate or defect as an atomic action. We propose to study the behaviors of online learning algorithms in the Iterated Prisoner's Dilemma (IPD) game, where we investigate the full spectrum of reinforcement learning agents: multi-armed bandits, contextual bandits and reinforcement learning. We evaluate them based on a tournament of iterated prisoner's dilemma where multiple agents can compete in a sequential fashion. This allows us to analyze the dynamics of policies learned by multiple self-interested independent reward-driven agents, and also allows us study the capacity of these algorithms to fit the human behaviors. Results suggest that considering the current situation to make decision is the worst in this kind of social dilemma game. Multiples discoveries on online learning behaviors and clinical validations are stated, as an effort to connect artificial intelligence algorithms with human behaviors and their abnormal states in neuropsychiatric conditions.

Motivation & Objective

• To study how online learning agents behave in social dilemma games like the Iterated Prisoner’s Dilemma (IPD).
• To evaluate whether attention to context improves reward maximization in IPD tournaments.
• To assess whether different online learning algorithms can accurately model human behavioral patterns in repeated social interactions.
• To explore connections between AI learning mechanisms and neuropsychiatric conditions through behavioral modeling.
• To bridge artificial intelligence and neuropsychiatry by analyzing how learning biases reflect human and pathological decision-making.

Proposed method

• Conducts a round-robin tournament of IPD with multiple agents, including pairwise, three-agent, and 'mental' agent variants.
• Employs three algorithm classes: multi-armed bandits (no context), contextual bandits (context-dependent), and reinforcement learning (state- and transition-dependent).
• Uses behavioral cloning to train algorithms on human decision data and test their predictive accuracy.
• Employs a reward-driven, self-interested learning framework where agents optimize cumulative payoff over repeated interactions.
• Applies empirical evaluation across different IPD settings to analyze policy dynamics and learning behaviors.
• Uses open-source code and data (available on GitHub) to ensure reproducibility of all results.

Experimental results

Research questions

• RQ1 Does attending to the current context improve reward maximization in the Iterated Prisoner’s Dilemma for online learning agents?
• RQ2 How do different online learning architectures—bandits, contextual bandits, and reinforcement learning—compare in terms of performance and behavioral dynamics in social dilemmas?
• RQ3 Can online learning algorithms effectively mimic human behavior in repeated social interactions?
• RQ4 What are the implications of these learning behaviors for understanding neuropsychiatric conditions involving reward processing and attentional biases?
• RQ5 How do different inductive biases in learning algorithms reflect or predict abnormal human decision-making patterns?

Key findings

• Contextual bandits perform poorly in IPD tournaments, indicating that focusing solely on the current state is suboptimal in social dilemma games.
• Multi-armed bandits (without context) achieve the best performance in maximizing rewards, suggesting that ignoring context can be more effective than partial context awareness.
• Bandit algorithms best fit empirical human behavioral data, implying that humans may not use contextual information when playing the IPD.
• The results challenge the assumption that context-aware decision-making is optimal in social dilemmas and suggest that over-attention to the present state can be detrimental.
• Simulation results align with clinical observations of reward processing deficits in conditions like ADHD, depression, and addiction, supporting the model's relevance to neuropsychiatric research.
• The study provides a foundation for using AI learning frameworks to model and understand human decision-making mechanisms and their dysfunctions.