[Paper Review] Guided Deep Reinforcement Learning for Swarm Systems

Maximilian Hüttenrauch, Adrian Šošić+1 more|arXiv (Cornell University)|Sep 18, 2017

Reinforcement Learning in Robotics6 references100 citations

TL;DR

The paper proposes an actor-critic framework where a centralized critic with access to the global state guides learning for homogeneous swarm agents, while each agent acts on local observations, enabling cooperative tasks like graph building and target localization in simulated swarms.

ABSTRACT

In this paper, we investigate how to learn to control a group of cooperative agents with limited sensing capabilities such as robot swarms. The agents have only very basic sensor capabilities, yet in a group they can accomplish sophisticated tasks, such as distributed assembly or search and rescue tasks. Learning a policy for a group of agents is difficult due to distributed partial observability of the state. Here, we follow a guided approach where a critic has central access to the global state during learning, which simplifies the policy evaluation problem from a reinforcement learning point of view. For example, we can get the positions of all robots of the swarm using a camera image of a scene. This camera image is only available to the critic and not to the control policies of the robots. We follow an actor-critic approach, where the actors base their decisions only on locally sensed information. In contrast, the critic is learned based on the true global state. Our algorithm uses deep reinforcement learning to approximate both the Q-function and the policy. The performance of the algorithm is evaluated on two tasks with simple simulated 2D agents: 1) finding and maintaining a certain distance to each others and 2) locating a target.

Motivation & Objective

• Address learning cooperative policies for swarms with limited sensing
• Leverage a centralized critic with full state during training while keeping actors locally observed
• Demonstrate end-to-end deep RL for homogeneous agents in two swarm tasks
• Evaluate scalability with varying swarm sizes and compare guided vs. non-guided learning

Proposed method

• Use an actor-critic framework with a centralized guided critic that evaluates joint actions using the global state
• Actors execute policies based on local observation histories, not the global state
• Represent the global swarm state with a compact vector to learn the Q-function while agents act on local observations
• Model the swarm as a swarm MDP with homogeneous agents sharing a single policy μ over histories H
• Incorporate a histogram-based fixed-size observation representation to handle varying neighbor counts
• Train with experience replay and slow target networks as in DDPG/DRL practice
• Provide two simulated tasks inspired by Kilobot: graph building (maintain distances to maximize edges) and target localization (cooperative search)
• Adopt a deterministic policy gradient for distributed local actors, and a centralized critic update using joint actions from μ(h^i) for all agents

Experimental results

Research questions

• RQ1 Can guided learning with a centralized critic enable effective policies for swarm systems with partial observability?
• RQ2 How does policy performance scale with the number of agents during training and evaluation?
• RQ3 Is guided learning necessary for swarm tasks, or can non-guided joint-history Q-learning succeed?
• RQ4 What is the impact of communication- and observation-modeling choices (histogram inputs) on learning outcomes?
• RQ5 How do learned policies compare across tasks (graph building vs localization) and agent counts?

Key findings

• Successful learning of distributed policies for 2–8 agents in both tasks, though harder with more agents
• Policies learned with guided critic outperform non-guided approaches in the edge (graph) task where joint history Q-learning failed
• In the localization task, policies with inter-agent communication outperform those without communication as agent count increases
• Different learned strategies emerge: small groups with circular motion for fewer agents versus larger aggregations for more agents
• The approach demonstrates end-to-end learning from high-dimensional sensory input to actions without hand-crafted features