[Paper Review] Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning

Elizaveta Tennant, Stephen Hailes+1 more|arXiv (Cornell University)|Jan 20, 2023

Evolutionary Game Theory and Cooperation4 citations

TL;DR

This paper introduces a novel framework for modeling moral decision-making in multi-agent reinforcement learning by embedding simplified moral reward functions based on utilitarianism, deontology, and virtue ethics. It demonstrates that norm-based and kindness-oriented moral agents learn to cooperate but are easily exploited, while equality-focused agents exhibit exploitative behavior during training, and mixed-virtue agents remain cooperative but vulnerable to exploitation.

ABSTRACT

Practical uses of Artificial Intelligence (AI) in the real world have demonstrated the importance of embedding moral choices into intelligent agents. They have also highlighted that defining top-down ethical constraints on AI according to any one type of morality is extremely challenging and can pose risks. A bottom-up learning approach may be more appropriate for studying and developing ethical behavior in AI agents. In particular, we believe that an interesting and insightful starting point is the analysis of emergent behavior of Reinforcement Learning (RL) agents that act according to a predefined set of moral rewards in social dilemmas. In this work, we present a systematic analysis of the choices made by intrinsically-motivated RL agents whose rewards are based on moral theories. We aim to design reward structures that are simplified yet representative of a set of key ethical systems. Therefore, we first define moral reward functions that distinguish between consequence- and norm-based agents, between morality based on societal norms or internal virtues, and between single- and mixed-virtue (e.g., multi-objective) methodologies. Then, we evaluate our approach by modeling repeated dyadic interactions between learning moral agents in three iterated social dilemma games (Prisoner's Dilemma, Volunteer's Dilemma and Stag Hunt). We analyze the impact of different types of morality on the emergence of cooperation, defection or exploitation, and the corresponding social outcomes. Finally, we discuss the implications of these findings for the development of moral agents in artificial and mixed human-AI societies.

Motivation & Objective

• To investigate how different moral frameworks—utilitarianism, deontology, and virtue ethics—shape the behavior of reinforcement learning agents in social dilemmas.
• To develop a systematic method for encoding moral reward functions that reflect key ethical systems in a simplified yet representative way.
• To analyze emergent cooperation, defection, and exploitation in dyadic interactions across three iterated social dilemma games.
• To explore the implications of moral diversity among AI agents for the design of ethical artificial and hybrid human-AI societies.
• To provide a foundational methodological platform for future research on moral agent interactions in complex, multi-agent environments.

Proposed method

• Designing moral reward functions based on three ethical frameworks: utilitarian (maximizing global reward), deontological (adhering to norms), and virtue ethics (emphasizing internal virtues like kindness and equality).
• Implementing intrinsically-motivated reinforcement learning agents whose policies are guided by these moral reward functions rather than purely self-interested payoffs.
• Evaluating agents in three iterated social dilemma games: Prisoner’s Dilemma, Volunteer’s Dilemma, and Stag Hunt, using repeated dyadic interactions.
• Using a multi-agent RL setup with fixed opponent types to analyze learning dynamics and social outcomes across 100 independent training runs.
• Introducing a mixed-virtue reward function that combines kindness and equality with adjustable weights to study multi-objective moral behavior.
• Analyzing convergence speed, cooperation rates, and exploitation patterns to compare moral agent types.

Experimental results

Research questions

• RQ1 How do different moral reward structures (utilitarian, deontological, virtue-based) influence the emergence of cooperation or defection in multi-agent RL?
• RQ2 What are the learning dynamics and social outcomes when moral agents interact in repeated social dilemmas?
• RQ3 How does a mixed-virtue moral agent (balancing kindness and equality) compare to pure virtue agents in terms of cooperation and vulnerability to exploitation?
• RQ4 To what extent do norm-based moral agents (e.g., deontological, virtue-kindness) learn cooperative policies that are nevertheless exploitable?
• RQ5 How does the weighting of different virtues in a multi-objective moral reward function affect agent behavior and social outcomes?

Key findings

• Utilitarian, deontological, and virtue-kindness agents consistently learned cooperative policies across all three games, achieving mutual cooperation in the Pareto-optimal outcome.
• These norm-based and kindness-oriented agents were systematically exploited by selfish opponents, indicating that cooperation does not guarantee resilience against exploitation.
• The virtue-equality agent exhibited exploitative behavior during the learning phase before convergence, suggesting that equality-focused moral systems may lead to less stable cooperation.
• The virtue-mixed agent, with equal weighting on kindness and equality, behaved similarly to the pure kindness agent—cooperative but vulnerable to exploitation—indicating that kindness dominates in the absence of strong equality incentives.
• Only when the weight on equality (β) was very high did the mixed-virtue agent shift toward more defensive and less cooperative behavior, showing that moral trade-offs significantly affect policy emergence.
• The study establishes a methodological foundation for modeling diverse moral agents in artificial societies, enabling future research on moral dynamics in complex, multi-agent environments.