[Paper Review] Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning

Jakub Grudzien Kuba, Ruiqing Chen+5 more|arXiv (Cornell University)|Sep 23, 2021

Reinforcement Learning in Robotics33 references83 citations

TL;DR

This paper extends trust region learning to cooperative multi-agent reinforcement learning by introducing a theoretical framework with monotonic improvement guarantees and proposing HATRPO and HAPPO algorithms that handle heterogeneous agents without parameter sharing.

ABSTRACT

Trust region methods rigorously enabled reinforcement learning (RL) agents to learn monotonically improving policies, leading to superior performance on a variety of tasks. Unfortunately, when it comes to multi-agent reinforcement learning (MARL), the property of monotonic improvement may not simply apply; this is because agents, even in cooperative games, could have conflicting directions of policy updates. As a result, achieving a guaranteed improvement on the joint policy where each agent acts individually remains an open challenge. In this paper, we extend the theory of trust region learning to MARL. Central to our findings are the multi-agent advantage decomposition lemma and the sequential policy update scheme. Based on these, we develop Heterogeneous-Agent Trust Region Policy Optimisation (HATPRO) and Heterogeneous-Agent Proximal Policy Optimisation (HAPPO) algorithms. Unlike many existing MARL algorithms, HATRPO/HAPPO do not need agents to share parameters, nor do they need any restrictive assumptions on decomposibility of the joint value function. Most importantly, we justify in theory the monotonic improvement property of HATRPO/HAPPO. We evaluate the proposed methods on a series of Multi-Agent MuJoCo and StarCraftII tasks. Results show that HATRPO and HAPPO significantly outperform strong baselines such as IPPO, MAPPO and MADDPG on all tested tasks, therefore establishing a new state of the art.

Motivation & Objective

• Motivate the extension of trust region methods to multi-agent settings where agents may have conflicting policy updates.
• Develop a theory that guarantees monotonic improvement for joint policies in cooperative MARL.
• Introduce a practical sequential update scheme and algorithms that do not require homogeneous agents or joint value function decomposability.
• Provide theoretical guarantees and empirical validation showing state-of-the-art performance on MARL benchmarks.

Proposed method

• Present multi-agent policy iteration with monotonic improvement via a multi-agent advantage decomposition lemma.
• Propose sequential per-agent policy updates that guarantee joint improvement (Algorithm 1).
• Derive theoretical results: monotonic improvement (Theorem 2) and convergence to Nash equilibria (Theorem 3).
• Introduce practical algorithms HATRPO and HAPPO that implement sequential updates with tractable KL constraints.
• Use a joint advantage estimator to compute updates without centralised critics for each agent (Proposition 2).
• Provide PPO-style clipping variant (HAPPO) to reduce computational burden while preserving performance.

Experimental results

Research questions

• RQ1 How can trust region theory be extended to cooperative MARL with heterogeneous agents?
• RQ2 Can a sequential per-agent update scheme guarantee monotonic improvement of the joint policy in MARL?
• RQ3 What are the theoretical convergence properties of such a scheme, and how can it be implemented efficiently in practice?
• RQ4 Do heterogeneous-agent TRPO/PPO algorithms achieve state-of-the-art performance on MARL benchmarks without shared parameters?
• RQ5 How can a joint advantage estimator be leveraged to avoid centralized critics while enabling reliable updates?

Key findings

• Introduces Multi-Agent Trust Region Learning with a monotonic improvement guarantee for cooperative MARL.
• Proves a multi-agent advantage decomposition enabling sequential updates that guarantee improvement (Theorem 2).
• Shows convergence to Nash equilibria under random update orders (Theorem 3).
• Presents HATRPO and HAPPO that do not require parameter sharing and do not assume joint Q-function decomposability.
• Demonstrates state-of-the-art performance on StarCraftII and Multi-Agent MuJoCo against MADDPG, IPPO, and MAPPO baselines.
• Offers a practical, scalable approach using joint advantage estimators to avoid multiple centralized critics.