[Paper Review] Regret Bounds for Decentralized Learning in Cooperative Multi-Agent Dynamical Systems

Seyed Mohammad Asghari, Yi Ouyang+1 more|arXiv (Cornell University)|Jan 27, 2020

Advanced Bandit Algorithms Research53 references4 citations

TL;DR

This paper proposes a decentralized multi-agent reinforcement learning (MARL) algorithm for cooperative linear-quadratic (LQ) dynamical systems with partial system knowledge and limited communication. By constructing an auxiliary single-agent LQ problem as an implicit coordination mechanism, the algorithm achieves a regret bound of $\tilde{O}(\sqrt{T})$, matching theoretical performance bounds in single-agent LQ control, even when only one agent has unknown dynamics and communication is one-directional.

ABSTRACT

Regret analysis is challenging in Multi-Agent Reinforcement Learning (MARL) primarily due to the dynamical environments and the decentralized information among agents. We attempt to solve this challenge in the context of decentralized learning in multi-agent linear-quadratic (LQ) dynamical systems. We begin with a simple setup consisting of two agents and two dynamically decoupled stochastic linear systems, each system controlled by an agent. The systems are coupled through a quadratic cost function. When both systems' dynamics are unknown and there is no communication among the agents, we show that no learning policy can generate sub-linear in $T$ regret, where $T$ is the time horizon. When only one system's dynamics are unknown and there is one-directional communication from the agent controlling the unknown system to the other agent, we propose a MARL algorithm based on the construction of an auxiliary single-agent LQ problem. The auxiliary single-agent problem in the proposed MARL algorithm serves as an implicit coordination mechanism among the two learning agents. This allows the agents to achieve a regret within $O(\sqrt{T})$ of the regret of the auxiliary single-agent problem. Consequently, using existing results for single-agent LQ regret, our algorithm provides a $ ilde{O}(\sqrt{T})$ regret bound. (Here $ ilde{O}(\cdot)$ hides constants and logarithmic factors). Our numerical experiments indicate that this bound is matched in practice. From the two-agent problem, we extend our results to multi-agent LQ systems with certain communication patterns.

Motivation & Objective

• To address the challenge of regret analysis in decentralized multi-agent reinforcement learning (MARL) under unknown system dynamics and limited communication.
• To establish fundamental limits on regret performance when both agents have unknown dynamics and no communication is available.
• To design a MARL algorithm that achieves sub-linear regret in the presence of partial system knowledge and asymmetric information flow.
• To extend the proposed framework to multi-agent LQ systems with structured communication patterns.

Proposed method

• Construct an auxiliary single-agent LQ problem to serve as an implicit coordination mechanism between two decentralized agents.
• Use one-directional communication from the agent controlling the unknown system to the other to enable coordinated learning.
• Define regret as the cumulative difference between the cost under the MARL policy and the optimal cost under full knowledge of system parameters.
• Leverage existing single-agent LQ regret bounds (e.g., from Abbasi-Yadkori & Szepesvári, 2011) to derive regret bounds for the MARL algorithm.
• Apply a transformation of the multi-agent system into a centralized auxiliary problem to decouple learning and control policy design.
• Use state estimation error dynamics and trace-based bounds to relate the regret of the MARL algorithm to that of the auxiliary single-agent problem.

Experimental results

Research questions

• RQ1 Can sub-linear regret be achieved in decentralized MARL when both agents have unknown system dynamics and no communication is allowed?
• RQ2 What is the fundamental regret limit in a two-agent cooperative LQ system with one-way communication and partial system knowledge?
• RQ3 How can an auxiliary single-agent LQ problem be used to implicitly coordinate two decentralized agents in a multi-agent system?
• RQ4 Can the regret of a decentralized MARL algorithm be bounded using the regret of a single-agent LQ learning algorithm?
• RQ5 How do structured communication patterns affect the regret performance in multi-agent LQ systems?

Key findings

• When both agents have unknown system dynamics and no communication is allowed, no learning policy can achieve sub-linear regret in $T$; regret is at best linear.
• With one-way communication from the agent controlling the unknown system, the proposed MARL algorithm achieves a regret bound of $\tilde{O}(\sqrt{T})$.
• The regret of the MARL algorithm is bounded by the regret of the auxiliary single-agent LQ problem plus a term that scales as $O(\sqrt{T})$ due to estimation errors.
• Numerical experiments confirm that the theoretical $\tilde{O}(\sqrt{T})$ regret bound is matched in practice.
• For multi-agent systems with two-way communication and unknown dynamics for both agents, the regret of the MARL algorithm equals that of the auxiliary single-agent LQ problem, achieving $R(T,\texttt{AL-MARL3}) = R^\diamond(T,\texttt{AL-SARL})$.
• The method generalizes to multi-agent LQ systems with specific communication patterns, maintaining $\tilde{O}(\sqrt{T})$ regret under appropriate assumptions.