[Paper Review] Survey of Self-Play in Reinforcement Learning

Anthony DiGiovanni, Ethan Zell|arXiv (Cornell University)|Jul 6, 2021

Game Theory and Applications19 references4 citations

TL;DR

This survey synthesizes self-play in reinforcement learning, analyzing its theoretical foundations, evaluation criteria, and algorithmic approaches. It identifies Pareto efficiency and egalitarian bargaining solutions as superior yet underexplored benchmarks, and highlights FCL as a novel algorithm that ensures self-play compatibility through coordinated exploration and punishment mechanisms for defection.

ABSTRACT

In reinforcement learning (RL), the term self-play describes a kind of multi-agent learning (MAL) that deploys an algorithm against copies of itself to test compatibility in various stochastic environments. As is typical in MAL, the literature draws heavily from well-established concepts in classical game theory and so this survey quickly reviews some fundamental concepts. In what follows, we present a brief survey of self-play literature, its major themes, criteria, and techniques, and then conclude with an assessment of current shortfalls of the literature as well as suggestions for future directions.

Motivation & Objective

• To clarify the theoretical and practical challenges in defining 'optimality' for self-play in multi-agent reinforcement learning.
• To evaluate competing performance benchmarks—such as Nash equilibrium, Pareto efficiency, and egalitarian bargaining solutions—within self-play frameworks.
• To analyze how algorithms maintain compatibility with self-copies while adapting to non-self agents, especially under uncertainty about reward functions.
• To identify open problems in finite-time guarantees, regret bounds, and robustness against adversarial or misaligned players.
• To explore the design of incentive-stable algorithms that prevent exploitation and ensure long-term cooperation in unknown or stochastic environments.

Proposed method

• Uses Markov games as the formal framework for modeling self-play, extending MDPs to include multiple agents with stochastic transitions and rewards.
• Proposes a three-component action-value estimation system: $Q^c$ (collective reward for EBS), $Q^r_j$ (punishment for deviating players), and $Q^d_j$ (deviation gains), enabling strategic response to non-cooperative behavior.
• Employs coordinated exploration across self-copies to ensure accurate estimation of $Q^c$, $Q^r_j$, and $Q^d_j$, preventing misinterpretation of exploration as defection.
• Introduces the Foolproof Cooperative Learning (FCL) algorithm, which uses surrogate reward functions to enforce the egalitarian bargaining solution (EBS) and deter deviations via credible punishment.
• Applies Q-learning with structured exploration schedules to balance exploration and reward maximization in environments with initially unknown dynamics.
• Uses game-theoretic concepts such as Nash equilibrium, Pareto efficiency, and the EBS to evaluate algorithmic performance and stability.

Experimental results

Research questions

• RQ1 What formal models of self-play are most theoretically tractable, and what proof techniques are commonly used to establish convergence?
• RQ2 How do different evaluation criteria—such as Nash equilibrium, Pareto efficiency, and egalitarian bargaining—compare in terms of practical desirability and theoretical feasibility?
• RQ3 How can algorithms maintain self-play compatibility while also guaranteeing safety and optimality against adversarial and stochastic non-self players?
• RQ4 What are the computational and theoretical tradeoffs in achieving higher standards of fairness and efficiency in self-play, such as the EBS versus a simple Nash equilibrium?
• RQ5 How can algorithms detect and respond to targeted adversaries or conflicting cooperative strategies to avoid destructive punishment loops?

Key findings

• The literature reveals significant divergence in performance criteria, with Pareto efficiency and egalitarian bargaining solutions (EBS) being more desirable but harder to achieve than simple convergence to any Nash equilibrium.
• Finite-time guarantees, particularly regret bounds, are largely absent in self-play algorithms, and existing bounds do not ensure high performance against non-self players.
• The FCL algorithm achieves Learning Equilibrium by maintaining three action-value estimates: $Q^c$ for EBS, $Q^r_j$ for punishment, and $Q^d_j$ for deviation incentives, enabling credible deterrence of defection.
• Coordinated exploration across self-copies ensures accurate estimation of $Q^c$, $Q^r_j$, and $Q^d_j$, preventing false positives in detecting defection and enabling convergence to the true EBS policy.
• Despite experimental success, M-Qubed lacks theoretical guarantees for balancing self-play efficiency and adaptability to non-self agents, raising concerns about robustness in complex or short-horizon settings.
• Punishment loops can emerge when two FCL agents pursue conflicting cooperative ideals, leading to mutual punishment and suboptimal outcomes, highlighting the need for additional criteria to detect and avoid such scenarios.