[Paper Review] Empirical Policy Optimization for $n$-Player Markov Games

Yuanheng Zhu, Dongbin Zhao+2 more|arXiv (Cornell University)|Oct 18, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes a distributed, history-accumulating policy optimization framework for n-player Markov games that provably converges to an approximate Nash equilibrium. By evolving policies based on aggregated historical performance rather than immediate rewards, the method ensures stability and convergence in non-stationary multi-agent environments, with empirical validation on small-scale games and a large-scale Pong environment using deep reinforcement learning.

ABSTRACT

In single-agent Markov decision processes, an agent can optimize its policy based on the interaction with environment. In multi-player Markov games (MGs), however, the interaction is non-stationary due to the behaviors of other players, so the agent has no fixed optimization objective. In this paper, we treat the evolution of player policies as a dynamical process and propose a novel learning scheme for Nash equilibrium. The core is to evolve one's policy according to not just its current in-game performance, but an aggregation of its performance over history. We show that for a variety of MGs, players in our learning scheme will provably converge to a point that is an approximation to Nash equilibrium. Combined with neural networks, we develop the \emph{empirical policy optimization} algorithm, that is implemented in a reinforcement-learning framework and runs in a distributed way, with each player optimizing its policy based on own observations. We use two numerical examples to validate the convergence property on small-scale MGs with $n\ge 2$ players, and a pong example to show the potential of our algorithm on large games.

Motivation & Objective

• To address the non-stationarity challenge in n-player Markov games, where each agent's optimization objective shifts due to others' evolving policies.
• To develop a learning scheme that enables convergence to a Nash equilibrium without requiring knowledge of other players' strategies or the number of players.
• To design a scalable, distributed reinforcement learning framework compatible with deep neural networks for large-scale games.
• To prove theoretical convergence of the proposed continuous-time learning dynamics to a Nash distribution using Lyapunov stability and fixed-point theory.

Proposed method

• Formulates a continuous-time learning dynamics (CTLD) where each player updates its policy based on the cumulative historical performance, not just current payoff.
• Applies Lyapunov stability and fixed-point theorems to prove that CTLD converges to a Nash distribution for a variety of n-player Markov games.
• Develops an empirical policy optimization (EPO) algorithm that uses neural networks to represent policies and trains parameters via reinforcement learning on full historical experience.
• Implements the EPO algorithm in a distributed manner, where each player acts solely on its own observations without requiring coordination or knowledge of others' strategies.
• Employs a replay buffer to store and utilize historical experience, enabling policy updates that avoid forgetting past effective strategies.
• Uses Schur decomposition for 2D visualization of policy populations to analyze convergence patterns and policy evolution.

Experimental results

Research questions

• RQ1 Can a policy optimization scheme in n-player Markov games converge to a Nash equilibrium despite non-stationary environments?
• RQ2 How does aggregating historical performance improve convergence compared to on-policy or self-play methods?
• RQ3 What is the theoretical guarantee of convergence for the proposed continuous-time learning dynamics in general n-player games?
• RQ4 How does the EPO algorithm perform in comparison to established baselines like PPO, NFSP, and PSRO in both small-scale and large-scale games?
• RQ5 To what extent does maintaining full historical experience prevent policy forgetting and improve equilibrium convergence?

Key findings

• The proposed continuous-time learning dynamics (CTLD) provably converges to a Nash distribution in a variety of n-player Markov games, as shown via Lyapunov stability and fixed-point analysis.
• Empirical Policy Optimization (EPO) outperforms PPO, Self-Play, NFSP, and PSRO in convergence speed and equilibrium quality, with relative performance curves dropping below baseline only for EPO in early training.
• EPO with full historical replay achieves monotonic or transitive policy improvement, while limited replay leads to cyclic behavior and policy forgetting, as visualized via Schur decomposition.
• In the Wimblepong environment, EPO with ε = 0.1 achieves superior performance compared to baselines, demonstrating scalability and effectiveness in large-scale games.
• The ablation study confirms that full historical experience is critical for stable convergence, as limited replay causes instability and suboptimal policy evolution.
• The distributed nature of EPO, requiring only local observations and no knowledge of other players’ strategies, makes it practical for real-world applications with limited coordination.