[Paper Review] Finite-Sample Analysis of Decentralized Q-Learning for Stochastic Games

Zuguang Gao, Qianqian Ma+2 more|arXiv (Cornell University)|Dec 15, 2021

Auction Theory and Applications4 citations

TL;DR

This paper provides finite-sample convergence guarantees for decentralized Q-learning in weakly acyclic stochastic games, establishing sample complexity bounds for both tabular and linear function approximation settings. It introduces a novel 'linear approximated equilibrium' concept and proves that decentralized Q-learning converges to this equilibrium with high probability under mild conditions, extending prior asymptotic results to non-asymptotic regimes.

ABSTRACT

Learning in stochastic games is arguably the most standard and fundamental setting in multi-agent reinforcement learning (MARL). In this paper, we consider decentralized MARL in stochastic games in the non-asymptotic regime. In particular, we establish the finite-sample complexity of fully decentralized Q-learning algorithms in a significant class of general-sum stochastic games (SGs) - weakly acyclic SGs, which includes the common cooperative MARL setting with an identical reward to all agents (a Markov team problem) as a special case. We focus on the practical while challenging setting of fully decentralized MARL, where neither the rewards nor the actions of other agents can be observed by each agent. In fact, each agent is completely oblivious to the presence of other decision makers. Both the tabular and the linear function approximation cases have been considered. In the tabular setting, we analyze the sample complexity for the decentralized Q-learning algorithm to converge to a Markov perfect equilibrium (Nash equilibrium). With linear function approximation, the results are for convergence to a linear approximated equilibrium - a new notion of equilibrium that we propose - which describes that each agent's policy is a best reply (to other agents) within a linear space. Numerical experiments are also provided for both settings to demonstrate the results.

Motivation & Objective

• To close the gap between asymptotic convergence and finite-sample performance in decentralized multi-agent reinforcement learning.
• To analyze the sample complexity of decentralized Q-learning in general-sum stochastic games, particularly weakly acyclic ones.
• To extend convergence guarantees to the linear function approximation setting, where full Q-function recovery is not guaranteed.
• To introduce and formalize the concept of a 'linear approximated equilibrium' as a practical alternative to Markov perfect equilibrium under function approximation.
• To provide explicit, game-parameter-dependent bounds on mixing time and stationary distribution minimum probability to make theoretical results actionable.

Proposed method

• Proposes a fully decentralized Q-learning algorithm (Algorithm 1) where each agent learns using only its own observations and local updates, without observing other agents’ actions or rewards.
• Introduces a new equilibrium concept—'linear approximated equilibrium'—where each agent’s policy is a best response within a linear function space of features.
• Uses a two-timescale update rule with time-varying learning rates to ensure convergence, where Q-value estimates are updated based on bootstrapped returns.
• Employs a novel analysis framework that bounds the minimum probability of visiting non-equilibrium policies and the mixing time of the Markov chain induced by joint policies.
• Derives closed-form lower bounds on the probability of policy update and convergence to equilibrium, using the minimum Bellman error as a key metric.
• Extends the analysis to linear function approximation (Algorithm 2), proving convergence to the linear approximated equilibrium with high probability under appropriate learning rate schedules.

Experimental results

Research questions

• RQ1 What is the finite-sample complexity of decentralized Q-learning in weakly acyclic stochastic games, and how does it depend on game parameters?
• RQ2 How can convergence be guaranteed in the presence of linear function approximation, where the true Q-function may not be representable?
• RQ3 What is the relationship between the linear approximated equilibrium and the classical Markov perfect equilibrium under function approximation?
• RQ4 How do mixing time and minimum stationary distribution probability affect the convergence rate in decentralized MARL?
• RQ5 Can explicit, game-parameter-dependent bounds be derived for key quantities like mixing time and stationary distribution to make theoretical results practical?

Key findings

• The paper establishes finite-sample convergence of decentralized Q-learning to a Markov perfect equilibrium in tabular weakly acyclic stochastic games, with explicit bounds on the number of steps required.
• For the linear function approximation case, the algorithm converges to a 'linear approximated equilibrium'—a new equilibrium concept where each agent's policy is optimal within the linear function space—under mild conditions.
• The convergence rate depends on the mixing time of the joint policy Markov chain and the minimum stationary distribution probability, both of which are bounded in terms of game parameters via Proposition 2.
• The analysis provides closed-form lower bounds on the probability of policy updates and convergence to equilibrium, which were not available in prior asymptotic analyses.
• Numerical experiments confirm that the fraction of time policies spend near equilibrium increases with both the number of episodes K and the number of steps T per episode.
• The results show that even with limited feature sets, the linear approximated equilibrium can closely align with the true Markov perfect equilibrium when the feature space is sufficiently rich.