[Paper Review] Non-cooperative Multi-agent Systems with Exploring Agents

Jalal Etesami, Christoph-Nikolas Straehle|arXiv (Cornell University)|May 25, 2020

Reinforcement Learning in Robotics59 references4 citations

TL;DR

This paper proposes a non-cooperative multi-agent reinforcement learning framework where agents use Boltzmann-Gibbs distributed strategies to explore near-optimal behaviors in Markov games. It establishes a unique solution to coupled Bellman equations under mild conditions and introduces provably convergent algorithms for finite and infinite horizon settings, linking the model to maximum causal entropy principles for joint goal and policy inference.

ABSTRACT

Multi-agent learning is a challenging problem in machine learning that has applications in different domains such as distributed control, robotics, and economics. We develop a prescriptive model of multi-agent behavior using Markov games. Since in many multi-agent systems, agents do not necessary select their optimum strategies against other agents (e.g., multi-pedestrian interaction), we focus on models in which the agents play "exploration but near optimum strategies". We model such policies using the Boltzmann-Gibbs distribution. This leads to a set of coupled Bellman equations that describes the behavior of the agents. We introduce a set of conditions under which the set of equations admit a unique solution and propose two algorithms that provably provide the solution in finite and infinite time horizon scenarios. We also study a practical setting in which the interactions can be described using the occupancy measures and propose a simplified Markov game with less complexity. Furthermore, we establish the connection between the Markov games with exploration strategies and the principle of maximum causal entropy for multi-agent systems. Finally, we evaluate the performance of our algorithms via several well-known games from the literature and some games that are designed based on real world applications.

Motivation & Objective

• To model multi-agent systems where agents explore near-optimal strategies without full cooperation or communication.
• To develop a game-theoretic framework based on Markov games that captures mutual awareness and stochastic strategy selection.
• To establish conditions under which the resulting coupled Bellman equations admit a unique solution, ensuring model stability.
• To design provably convergent algorithms for both finite and infinite time horizon scenarios.
• To connect the model to maximum causal entropy principles for joint inference of agent goals and policies.

Proposed method

• Models agent strategies using the Boltzmann-Gibbs distribution, where action probabilities depend on Q-values and a temperature parameter.
• Derives a system of coupled Bellman equations that jointly determine Q-values and policies, reflecting mutual dependence between agents.
• Introduces sufficient conditions (e.g., contraction mapping properties) ensuring a unique solution to the coupled equations.
• Proposes two algorithms: one for finite-horizon and one for infinite-horizon settings, both proven to converge to the unique solution.
• Introduces a simplified forward-backward algorithm using occupancy measures to reduce computational complexity in practical settings.
• Establishes a formal link between the proposed model and the principle of maximum causal entropy, enabling joint inference of goals and policies.

Experimental results

Research questions

• RQ1 Under what conditions does the system of coupled Bellman equations for multi-agent exploration have a unique solution?
• RQ2 How can agents in a non-cooperative, communication-free environment select exploration strategies that are both near-optimal and consistent with mutual awareness?
• RQ3 Can the proposed algorithms guarantee convergence to the solution in both finite and infinite time horizon scenarios?
• RQ4 How can occupancy measures be leveraged to reduce the computational complexity of solving the Markov game?
• RQ5 What is the theoretical connection between the proposed model and the principle of maximum causal entropy in multi-agent systems?

Key findings

• The system of coupled Bellman equations admits a unique solution under mild conditions, including contraction mapping properties on the joint policy update.
• The proposed finite-horizon algorithm converges to the unique solution in finite time, with convergence guaranteed by the contraction mapping principle.
• The infinite-horizon algorithm also converges to the unique solution, with convergence proven via fixed-point iteration under appropriate assumptions.
• The forward-backward algorithm based on occupancy measures reduces complexity compared to the general Markov game formulation, enabling scalable inference.
• The model is formally connected to the principle of maximum causal entropy, allowing agents to jointly infer each other’s goals and strategies from observed behavior.
• Empirical evaluation on benchmark and real-world-inspired games confirms the model’s ability to generate stable, near-optimal, and collision-avoiding behaviors in competitive settings.