[Paper Review] Gradient play in stochastic games: stationary points, convergence, and sample complexity

Runyu Zhang, Zhaolin Ren+1 more|arXiv (Cornell University)|Jun 1, 2021

Reinforcement Learning in Robotics45 references4 citations

TL;DR

This paper establishes the equivalence between first-order stationary points and Nash equilibria (NEs) in stochastic games with directly parameterized policies, proving local convergence of gradient play to strict NEs and providing a non-asymptotic global convergence rate for Markov potential games. It further designs a sample-based decentralized reinforcement learning algorithm with a sample complexity of $\widetilde{O}\left(\frac{n}{\epsilon^{6}}\textup{poly}\left(\frac{1}{1-\gamma},|\mathcal{S}|,\max_{i}|\mathcal{A}_{i}|\right)\right)$ to reach an $\epsilon$-NE.

ABSTRACT

We study the performance of the gradient play algorithm for stochastic games (SGs), where each agent tries to maximize its own total discounted reward by making decisions independently based on current state information which is shared between agents. Policies are directly parameterized by the probability of choosing a certain action at a given state. We show that Nash equilibria (NEs) and first-order stationary policies are equivalent in this setting, and give a local convergence rate around strict NEs. Further, for a subclass of SGs called Markov potential games (which includes the setting with identical rewards as an important special case), we design a sample-based reinforcement learning algorithm and give a non-asymptotic global convergence rate analysis for both exact gradient play and our sample-based learning algorithm. Our result shows that the number of iterations to reach an $ε$-NE scales linearly, instead of exponentially, with the number of agents. Local geometry and local stability are also considered, where we prove that strict NEs are local maxima of the total potential function and fully-mixed NEs are saddle points.

Motivation & Objective

• To understand the relationship between first-order stationary points and Nash equilibria (NEs) in stochastic games with parameterized policies.
• To analyze the local convergence behavior of gradient play around strict NEs and characterize their stability.
• To design a sample-efficient, fully decentralized reinforcement learning algorithm for Markov potential games with global convergence guarantees.
• To quantify the sample complexity of learning an $\epsilon$-Nash equilibrium in stochastic games with finite state-action spaces.

Proposed method

• Generalizes the gradient domination property from single-agent to multi-agent stochastic games to establish equivalence between first-order stationary policies and NEs.
• Analyzes local convergence of gradient play to strict NEs using local geometry and stability analysis.
• Introduces the concept of Markov potential games (MPGs), a subclass of stochastic games that includes identical-reward settings, and proves global convergence to NE under gradient play.
• Proposes a sample-based, fully-decentralized reinforcement learning algorithm that leverages averaged MDPs for each agent when others' policies are fixed.
• Uses model-based policy evaluation with respect to agents' averaged MDPs to estimate gradients from samples.
• Employs a non-asymptotic analysis to derive a sample complexity bound scaling linearly with the number of agents.

Experimental results

Research questions

• RQ1 Are first-order stationary points of the gradient play algorithm equivalent to Nash equilibria in stochastic games with parameterized policies?
• RQ2 What is the local stability and convergence behavior of gradient play around strict Nash equilibria?
• RQ3 Can global convergence to a Nash equilibrium be guaranteed for gradient play in Markov potential games?
• RQ4 What is the sample complexity of learning an $\epsilon$-Nash equilibrium in stochastic games using a decentralized, sample-based algorithm?
• RQ5 How does the number of agents affect the convergence rate and sample efficiency of the learning algorithm?

Key findings

• First-order stationary policies are equivalent to Nash equilibria in stochastic games with directly parameterized policies, establishing a fundamental link between optimization and game-theoretic equilibria.
• Gradient play locally converges to strict Nash equilibria in finite steps, with a local convergence rate established for this setting.
• Strict Nash equilibria are local maxima of the total potential function, indicating their stability under gradient play dynamics.
• Fully mixed Nash equilibria are saddle points under gradient play, implying instability and potential divergence from such points.
• The proposed sample-based reinforcement learning algorithm achieves $\epsilon$-Nash equilibrium with high probability using $\widetilde{O}\left(\frac{n}{\epsilon^{6}}\textup{poly}\left(\frac{1}{1-\gamma},|\mathcal{S}|,\max_{i}|\mathcal{A}_{i}|\right)\right)$ samples.
• The sample complexity scales linearly with the number of agents $n$, indicating favorable scalability compared to exponential scaling in prior results.