[Paper Review] Learning Nash Equilibrium for General-Sum Markov Games from Batch Data

Julien Pérolat, Florian Strub+2 more|arXiv (Cornell University)|Jun 28, 2016

Game Theory and Applications28 citations

TL;DR

This paper proposes a novel method to learn an ǫ-Nash equilibrium in general-sum Markov Games from batch data using function approximation. It introduces a weak ǫ-Nash equilibrium concept and proves that minimizing the Lp norm of two Bellman-like residuals ensures convergence to such an equilibrium, which is then optimized via a neural network architecture (NashNetwork) that jointly learns value functions and policies, achieving stable, scalable learning across multiplayer settings with empirical validation on synthetic Garnet games.

ABSTRACT

This paper addresses the problem of learning a Nash equilibrium in $\gamma$-discounted multiplayer general-sum Markov Games (MG). A key component of this model is the possibility for the players to either collaborate or team apart to increase their rewards. Building an artificial player for general-sum MGs implies to learn more complex strategies which are impossible to obtain by using techniques developed for two-player zero-sum MGs. In this paper, we introduce a new definition of $\epsilon$-Nash equilibrium in MGs which grasps the strategy's quality for multiplayer games. We prove that minimizing the norm of two Bellman-like residuals implies the convergence to such an $\epsilon$-Nash equilibrium. Then, we show that minimizing an empirical estimate of the $L_p$ norm of these Bellman-like residuals allows learning for general-sum games within the batch setting. Finally, we introduce a neural network architecture named NashNetwork that successfully learns a Nash equilibrium in a generic multiplayer general-sum turn-based MG.

Motivation & Objective

• Address the challenge of learning Nash equilibria in multiplayer general-sum Markov Games when the environment dynamics and rewards are unknown.
• Overcome the limitations of existing methods that fail in general-sum settings by introducing a weak ǫ-Nash equilibrium concept.
• Enable function approximation in batch settings to scale to large state spaces where tabular methods fail.
• Develop a unified deep learning framework that jointly optimizes value functions and policies for all players.
• Demonstrate scalability and robustness across varying numbers of players in synthetic, deterministic Markov Games.

Proposed method

• Introduce a new definition of weak ǫ-Nash equilibrium tailored for multiplayer general-sum Markov Games.
• Define two Bellman-like residuals—one for value function consistency and one for policy optimality—whose joint minimization ensures ǫ-Nash equilibrium convergence.
• Formulate an empirical Lp norm objective over these residuals using batch data, enabling off-policy learning without online interaction.
• Propose a novel neural network architecture, NashNetwork, with shared Q-networks and policy networks per player, using ReLU and softmax activations.
• Train the network via AdamGrad with separate learning rates and weight decay, minimizing the empirical Bellman residual norm on training and test sets.
• Use a binary state encoding and a circular reward function in synthetic Garnet environments to evaluate generalization and scalability.

Experimental results

Research questions

• RQ1 Can a weak ǫ-Nash equilibrium be formally defined and characterized in general-sum multiplayer Markov Games?
• RQ2 Does minimizing the sum of two Bellman-like residuals guarantee convergence to a weak ǫ-Nash equilibrium in such games?
• RQ3 Can function approximation be effectively combined with batch data to learn Nash equilibria in large-scale multiplayer Markov Games?
• RQ4 How does the proposed NashNetwork architecture scale across different numbers of players in synthetic environments?
• RQ5 To what extent does the empirical Bellman residual norm correlate with the actual error in best response, indicating equilibrium quality?

Key findings

• The method successfully learns a weak ǫ-Nash equilibrium in multiplayer general-sum Markov Games using only batch data, with convergence observed in training and test metrics.
• The Error vs Best Response metric dropped close to zero across all players, indicating that no player could improve their cumulative reward by more than 8% on average by unilaterally changing strategy.
• The strategy quality remained well-balanced among players, with a standard deviation of less than 5 points across the state space, showing consistent performance.
• The neural network scaled effectively from 2 to 5 players, maintaining similar strategy quality, demonstrating robustness to increasing player count.
• The empirical Bellman residual norm decreased in parallel with the Error vs Best Response metric, confirming that the training objective aligns with equilibrium quality.
• Overfitting was observed with insufficient data, and convergence required a high number of samples, suggesting that data efficiency remains a challenge for the method.