[Paper Review] Convergence of Multi-Agent Learning with a Finite Step Size in General-Sum Games

Xinliang Song, Tonghan Wang+1 more|arXiv (Cornell University)|Mar 7, 2019

Reinforcement Learning in Robotics24 references4 citations

TL;DR

This paper proposes GA-SPP, a gradient-based multi-agent learning algorithm that uses shrinking policy prediction to enable Nash convergence with a finite step size in general-sum games. Unlike prior methods, it guarantees convergence to Nash equilibrium in $m\times n$ positive semi-definite games, $2\times n$ games, and $2\times 2$ games without requiring infinitesimal learning rates.

ABSTRACT

Learning in a multi-agent system is challenging because agents are simultaneously learning and the environment is not stationary, undermining convergence guarantees. To address this challenge, this paper presents a new gradient-based learning algorithm, called Gradient Ascent with Shrinking Policy Prediction (GA-SPP), which augments the basic gradient ascent approach with the concept of shrinking policy prediction. The key idea behind this algorithm is that an agent adjusts its strategy in response to the forecasted strategy of the other agent, instead of its current one. GA-SPP is shown formally to have Nash convergence in larger settings than existing gradient-based multi-agent learning methods. Furthermore, unlike existing gradient-based methods, GA-SPP's theoretical guarantees do not assume the learning rate to be infinitesimal.

Motivation & Objective

• Address the challenge of non-stationary environments in multi-agent learning where agents' simultaneous learning undermines convergence.
• Overcome the limitation of existing gradient-based methods that require infinitesimal learning rates for theoretical convergence.
• Provide stronger theoretical guarantees for convergence to Nash equilibrium in broader classes of general-sum games.
• Develop a practical and theoretically grounded algorithm that improves upon existing policy prediction-based approaches like IGA-PP and extra-gradient methods.
• Enable more robust and scalable multi-agent learning in complex environments by relaxing assumptions on learning rate and strategy projection.

Proposed method

• Introduce a novel gradient ascent algorithm, Gradient Ascent with Shrinking Policy Prediction (GA-SPP), which forecasts the opponent's strategy and adjusts its policy based on this forecast.
• Use shrinking prediction lengths that can differ from the policy update rate, allowing greater flexibility and stronger theoretical guarantees.
• Project forecasted opponent strategies onto the valid probability simplex at each step to ensure strategy validity and support Nash convergence.
• Formally prove Nash convergence for GA-SPP in $m\times n$ positive semi-definite games, a subclass of $2\times n$ general-sum games, and $2\times 2$ general-sum games.
• Leverage convex projection operators $\Pi_{\bm{\Delta}}$ and $P_{\bm{\Delta}}(\bm{x}, \bm{v})$ to maintain strategy validity during updates.
• Integrate a finite learning rate $\eta$ without sacrificing convergence guarantees, breaking the prior assumption of infinitesimal step sizes.

Experimental results

Research questions

• RQ1 Can a gradient-based multi-agent learning algorithm achieve Nash convergence with a finite learning rate in general-sum games?
• RQ2 In which classes of general-sum games does GA-SPP guarantee convergence to a Nash equilibrium?
• RQ3 How does shrinking policy prediction improve convergence stability and theoretical guarantees compared to existing methods like IGA-PP and extra-gradient?
• RQ4 Does the projection of predicted strategies onto the valid strategy space enhance convergence to Nash equilibrium?
• RQ5 Can GA-SPP outperform existing algorithms like GIGA-WoLF and IGA-PP in empirical settings beyond the theoretical scope?

Key findings

• GA-SPP is the first gradient-ascent-based multi-agent learning algorithm with finite step size that guarantees Nash convergence in $m\times n$ positive semi-definite games.
• GA-SPP ensures convergence to a Nash equilibrium in a broader class of games than previous methods, including $2\times n$ general-sum games and $2\times 2$ games.
• Empirical results show GA-SPP converges to Nash equilibrium in benchmark games such as Prisoner’s Dilemma, Chicken, Battle of the Sexes, and Rock-Paper-Scissors.
• In Shapley’s game and a $2\times 3$ game, GA-SPP converges to Nash equilibrium where GIGA-WoLF fails, and outperforms IGA-PP in convergence stability.
• GA-SPP maintains convergence to Nash equilibrium across varying prediction lengths, unlike IGA-PP, which is sensitive to prediction length and may converge to non-equilibrium strategies.
• In three-player matching pennies, GA-SPP fails to converge with constant prediction length, indicating limitations in highly non-linear or chaotic dynamics.