[Paper Review] Coordinated Exploration in Concurrent Reinforcement Learning

Maria Dimakopoulou, Benjamin Van Roy|arXiv (Cornell University)|Feb 5, 2018

Advanced Bandit Algorithms Research12 references3 citations

TL;DR

This paper proposes seed sampling, a novel extension of posterior sampling for reinforcement learning that enables efficient coordinated exploration in multi-agent concurrent learning. By sampling a shared seed to generate MDPs, agents maintain commitment to exploration paths while adapting to shared data and ensuring behavioral diversity, significantly reducing per-agent regret compared to Thompson resampling and UCRL-based methods.

ABSTRACT

We consider a team of reinforcement learning agents that concurrently learn to operate in a common environment. We identify three properties - adaptivity, commitment, and diversity - which are necessary for efficient coordinated exploration and demonstrate that straightforward extensions to single-agent optimistic and posterior sampling approaches fail to satisfy them. As an alternative, we propose seed sampling, which extends posterior sampling in a manner that meets these requirements. Simulation results investigate how per-agent regret decreases as the number of agents grows, establishing substantial advantages of seed sampling over alternative exploration schemes.

Motivation & Objective

• To address the limitations of naive extensions of posterior sampling and upper-confidence-bound methods in multi-agent reinforcement learning.
• To identify and satisfy three key properties for efficient exploration: adaptivity to shared observations, commitment to exploration sequences, and behavioral diversity across agents.
• To design a scalable, robust exploration mechanism that outperforms existing concurrent RL algorithms in terms of regret minimization.
• To demonstrate that seed sampling enables faster convergence and superior sample efficiency in multi-agent environments.

Proposed method

• Agents sample a shared random seed at the start of learning, which is used to generate a stochastic MDP model from the current posterior distribution.
• Each agent's MDP is constructed using the seed and a deterministic mapping, ensuring consistent exploration paths across time while preserving diversity through independent seed sampling.
• The method maintains commitment by fixing the MDP model throughout an episode, avoiding disruptive resampling at each time step.
• Adaptivity is achieved by updating the posterior over MDP parameters using data collected by all agents, which informs future seed-based MDP sampling.
• Two specific seed sampling variants are proposed: standard-Gaussian and martingalean-Gaussian, tailored to different prior and likelihood assumptions.
• The approach generalizes PSRL to concurrent settings by decoupling exploration diversity from time-dependent randomness, enabling stable and coordinated learning.

Experimental results

Research questions

• RQ1 How can multi-agent reinforcement learning achieve efficient exploration when agents learn concurrently and share data?
• RQ2 Why do standard extensions of posterior sampling and UCB fail to coordinate exploration effectively in concurrent settings?
• RQ3 What properties are essential for coordinated exploration to reduce regret in multi-agent RL?
• RQ4 Can a seeding mechanism improve regret decay rates compared to Thompson resampling and concurrent UCRL?
• RQ5 How does seed sampling maintain adaptivity, commitment, and diversity simultaneously in concurrent learning?

Key findings

• Seed sampling achieves the lowest cumulative regret among all evaluated algorithms in the maximum reward path problem, with 10,000 agents and H=10 horizon.
• Concurrent UCRL incurs 49.9% higher cumulative regret than seed sampling due to lack of behavioral diversity, despite satisfying commitment.
• Thompson resampling incurs 191% higher cumulative regret than seed sampling due to lack of commitment and disruptive resampling.
• The regret decay rate of seed sampling is robust across problem instances, while alternative methods show significantly slower decay in certain environments.
• Standard-Gaussian and martingalean-Gaussian seed sampling variants both outperform alternatives, confirming the effectiveness of the seeding mechanism.
• Simulation results demonstrate that seed sampling enables agents to explore diverse paths while maintaining consistency, accelerating convergence to optimal policies.