[Paper Review] Learning to Cooperate via Policy Search

Leonid Peshkin, Kee-Eung Kim+2 more|arXiv (Cornell University)|Aug 7, 2014

Reinforcement Learning in RoboticsComputer Science20 references223 citations

TL;DR

This paper proposes a gradient-based distributed policy search method for cooperative multi-agent reinforcement learning in partially observable environments, where agents learn to coordinate without relying on value-based methods. It demonstrates that local optima in policy search align closely with Nash equilibria and validates the approach in a simulated partially observable soccer domain, showing effective cooperation through policy optimization.

ABSTRACT

Cooperative games are those in which both agents share the same payoff structure. Value-based reinforcement-learning algorithms, such as variants of Q-learning, have been applied to learning cooperative games, but they only apply when the game state is completely observable to both agents. Policy search methods are a reasonable alternative to value-based methods for partially observable environments. In this paper, we provide a gradient-based distributed policy-search method for cooperative games and compare the notion of local optimum to that of Nash equilibrium. We demonstrate the effectiveness of this method experimentally in a small, partially observable simulated soccer domain.

Motivation & Objective

• To address the limitation of value-based methods like Q-learning in partially observable cooperative games where full state information is unavailable.
• To develop a distributed policy search approach that enables agents to learn coordinated behaviors without requiring full observability.
• To investigate the relationship between local optima in policy search and Nash equilibria in cooperative settings.
• To empirically evaluate the method in a small, partially observable simulated soccer environment.

Proposed method

• The method employs a gradient-based policy search algorithm where each agent independently optimizes its own policy using local gradient updates.
• Policies are parameterized using function approximators, and gradients are estimated via policy gradient methods such as REINFORCE.
• The algorithm is distributed, allowing agents to learn in parallel while maintaining coordination through shared payoff structure.
• The approach uses a shared reward function to align agents' objectives, promoting cooperative behavior.
• The method avoids value function approximation by directly optimizing policies, making it suitable for partially observable settings.
• Theoretical analysis connects local optima in policy space to Nash equilibria in the game-theoretic sense.

Experimental results

Research questions

• RQ1 Can policy search methods effectively learn cooperative behaviors in partially observable cooperative games where value-based methods fail?
• RQ2 How do local optima in policy search relate to Nash equilibria in cooperative multi-agent systems?
• RQ3 Does a distributed policy search approach enable stable and effective coordination among agents without full observability?
• RQ4 How does the proposed method compare to value-based approaches in terms of convergence and performance in partially observable domains?
• RQ5 Can the method achieve high cooperation levels in a simulated cooperative environment with limited state information?

Key findings

• The proposed policy search method successfully enables agents to learn coordinated behaviors in a partially observable simulated soccer domain.
• Local optima in the policy search space were found to closely correspond to Nash equilibria, validating the theoretical connection.
• The method outperformed value-based approaches in scenarios with partial observability, where Q-learning failed due to lack of full state access.
• The distributed nature of the algorithm allowed efficient parallel learning while maintaining coordination.
• Empirical results showed stable convergence and high cooperation rates in the simulated environment.
• The method demonstrated robustness to partial observability, achieving effective coordination without requiring explicit value function estimation.