[Paper Review] Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning

Shariq Iqbal, Fei Sha|arXiv (Cornell University)|May 28, 2019

Reinforcement Learning in Robotics34 references34 citations

TL;DR

The paper introduces intrinsic rewards that coordinate exploration among multiple agents, learns multiple exploration policies in parallel with a shared replay buffer, and uses a meta-policy to dynamically select among them to maximize extrinsic rewards in sparse-cooperative tasks.

ABSTRACT

Solving tasks with sparse rewards is one of the most important challenges in reinforcement learning. In the single-agent setting, this challenge is addressed by introducing intrinsic rewards that motivate agents to explore unseen regions of their state spaces; however, applying these techniques naively to the multi-agent setting results in agents exploring independently, without any coordination among themselves. Exploration in cooperative multi-agent settings can be accelerated and improved if agents coordinate their exploration. In this paper we introduce a framework for designing intrinsic rewards which consider what other agents have explored such that the agents can coordinate. Then, we develop an approach for learning how to dynamically select between several exploration modalities to maximize extrinsic rewards. Concretely, we formulate the approach as a hierarchical policy where a high-level controller selects among sets of policies trained on diverse intrinsic rewards and the low-level controllers learn the action policies of all agents under these specific rewards. We demonstrate the effectiveness of the proposed approach in cooperative domains with sparse rewards where state-of-the-art methods fail and challenging multi-stage tasks that necessitate changing modes of coordination.

Motivation & Objective

• Address sparse-reward cooperative MARL by enabling coordinated exploration across agents.
• Propose intrinsic reward functions that account for other agents’ explored regions.
• Learn multiple exploration policies in parallel using a shared replay buffer for sample efficiency.
• Develop a meta-policy to dynamically select among exploration modalities to maximize extrinsic rewards.

Proposed method

• Define multi-agent intrinsic rewards that depend on how novel observations are across all agents.
• Describe desiderata for coordination-friendly intrinsic rewards (coordinate-wise monotonicity and inner-directedness).
• Train several policies in parallel, each trained with a different intrinsic reward type, using a shared replay buffer and off-policy SAC variants.
• Use a hierarchical setup where a high-level meta-policy selects among policies trained with different intrinsic rewards to maximize extrinsic returns.
• Formulate policy gradients and Q-function targets for both extrinsic and intrinsic rewards, with a cross-head architecture sharing bases across reward types.
• Train a meta-policy with entropy to maintain exploration across reward types.

Experimental results

Research questions

• RQ1 Does coordinating exploration via multi-agent intrinsic rewards improve performance on sparse-reward cooperative tasks?
• RQ2 Can a meta-policy effectively switch between diverse exploration modalities to maximize extrinsic rewards?
• RQ3 Do adaptive, coordinated exploration strategies outperform fixed or naive adaptations in multi-agent settings?
• RQ4 How does sharing data via a common replay buffer affect sample efficiency across multiple intrinsic reward types?

Key findings

• Multi-agent intrinsic rewards that coordinate across agents outperform independent intrinsic exploration in several tasks.
• The adaptive approach matches or exceeds the non-adaptive oracle (best fixed reward type) using the same sample budget.
• In tasks requiring changing coordination strategies, the adaptive method outperforms all fixed reward functions.
• The meta-policy balances exploration and exploitation and prevents collapse to a single exploration type.
• Adaptive strategies show advantages in tasks with varying coordination needs (e.g., Task 3 and Flip-Task).
• The method remains effective in both gridworld and VizDoom domains.