[Paper Review] Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning

Ying Wen, Yaodong Yang+3 more|arXiv (Cornell University)|Jan 26, 2019

Reinforcement Learning in RoboticsComputer Science51 citations

TL;DR

Introduces PR2, a probabilistic recursive reasoning framework for multi-agent deep RL, using variational Bayes to model opponents' conditional policies and derive decentralized PR2-Q and PR2-AC with convergence guarantees in self-play.

ABSTRACT

Humans are capable of attributing latent mental contents such as beliefs or intentions to others. The social skill is critical in daily life for reasoning about the potential consequences of others' behaviors so as to plan ahead. It is known that humans use such reasoning ability recursively by considering what others believe about their own beliefs. In this paper, we start from level-$1$ recursion and introduce a probabilistic recursive reasoning (PR2) framework for multi-agent reinforcement learning. Our hypothesis is that it is beneficial for each agent to account for how the opponents would react to its future behaviors. Under the PR2 framework, we adopt variational Bayes methods to approximate the opponents' conditional policies, to which each agent finds the best response and then improve their own policies. We develop decentralized-training-decentralized-execution algorithms, namely PR2-Q and PR2-Actor-Critic, that are proved to converge in the self-play scenarios when there exists one Nash equilibrium. Our methods are tested on both the matrix game and the differential game, which have a non-trivial equilibrium where common gradient-based methods fail to converge. Our experiments show that it is critical to reason about how the opponents believe about what the agent believes. We expect our work to contribute a new idea of modeling the opponents to the multi-agent reinforcement learning community.

Motivation & Objective

• Motivate the use of recursive reasoning to model how opponents react to an agent's future actions.
• Propose a probabilistic framework (PR2) that accounts for opponents' beliefs about the agent via a learned conditional policy.
• Develop decentralized-training-decentralized-execution algorithms (PR2-Q and PR2-AC) based on this framework.
• Provide theoretical convergence guarantees in self-play when a single Nash equilibrium exists.
• Demonstrate improved performance in matrix games, differential games, and particle world environments over baselines.

Proposed method

• Model the joint policy with a level-1 recursive decomposition that captures how opponents would respond to the agent's action.
• Approximate the opponents' conditional policy using variational inference, denoted as rho^{-i}_{phi^{-i}}(a^{-i}|s,a^{i}).
• Derive a multi-agent policy gradient that incorporates the expected Q-value under the opponents' conditional policy (PR2-GD update).
• Provide a decentralized-training-with-decentralized-execution algorithm (PR2-AC and PR2-Q) that does not require access to opponents’ policy parameters.
• Prove convergence of PR2 in self-play when there is a unique Nash equilibrium via a contraction operator for the PR2 soft value iteration.
• Employ amortized Stein Variational Gradient Descent (SVGD) to sample from the opponent conditional policy in continuous action spaces.

Experimental results

Research questions

• RQ1 Can recursive reasoning about opponents' beliefs improve learning in multi-agent RL beyond non-correlated factorization?
• RQ2 How can variational inference be used to model opponents' conditional policies in a tractable, decentralized training setting?
• RQ3 Do PR2-Q and PR2-AC converge to equilibria in self-play when a single Nash equilibrium exists?
• RQ4 Do PR2 methods outperform standard baselines in matrix games, differential games, and particle world environments?
• RQ5 What is the impact of reasoning about opponents' beliefs on exploration and convergence in continuous action spaces?

Key findings

• PR2 enables agents to account for how opponents would respond to their actions, leading to improved learning outcomes compared to baselines.
• PR2-Q and PR2-AC converge in self-play scenarios when there exists one Nash equilibrium.
• In the iterated matrix game, PR2 avoids non-convergent rotational dynamics observed by Infinitesimal Gradient Ascent and reaches the central equilibrium.
• In the Max of Two Quadratics differential game, PR2-AC converges to the global equilibrium while many baselines get trapped in local optima.
• PR2 methods show superior performance in cooperative and competitive settings within Particle World environments, especially in decentralized execution.
• Variational inference provides a practical means to approximate opponents' conditional policies, enabling scalable multi-agent reasoning.