[Paper Review] Graph Policy Gradients for Large Scale Unlabeled Motion Planning with Constraints

Arbaaz Khan, Vijay Kumar+1 more|arXiv (Cornell University)|Sep 24, 2019

Multimodal Machine Learning Applications15 references4 citations

TL;DR

This paper proposes Graph Policy Gradients (GPG), a scalable reinforcement learning framework for large-scale unlabeled motion planning with constraints. By using graph convolutional networks (GCNs) to parametrize decentralized policies, GPG enables zero-shot transfer from small to large robot swarms, achieving near-optimal performance within ~12–15 seconds of a centralized oracle solution across varying swarm sizes and goal configurations.

ABSTRACT

In this paper, we present a learning method to solve the unlabelled motion problem with motion constraints and space constraints in 2D space for a large number of robots. To solve the problem of arbitrary dynamics and constraints we propose formulating the problem as a multi-agent problem. In contrast to previous works that propose using learning solutions for unlabelled motion planning with constraints, we are able to demonstrate the scalability of our methods for a large number of robots. The curse of dimensionality one encounters when working with a large number of robots is mitigated by employing a graph convolutional neural (GCN) network to parametrize policies for the robots. The GCN reduces the dimensionality of the problem by learning filters that aggregate information among robots locally, similar to how a convolutional neural network is able to learn local features in an image. Additionally, by employing a GCN we are also able to overcome the computational overhead of training policies for a large number of robots by first training graph filters for a small number of robots followed by zero-shot policy transfer to a larger number of robots. We demonstrate the effectiveness of our framework through various simulations.

Motivation & Objective

• Address the scalability challenge in unlabeled multi-robot motion planning with dynamics and spatial constraints.
• Overcome the curse of dimensionality and computational overhead in training policies for large numbers of robots.
• Enable decentralized, local coordination among robots using graph-structured information aggregation.
• Achieve near-optimal performance compared to centralized, provably correct methods while maintaining scalability.
• Demonstrate zero-shot policy transfer from small to large robot swarms using learned graph filters.

Proposed method

• Formulate the unlabelled motion planning problem as a multi-agent reinforcement learning (MARL) task with a graph-structured robot interaction topology.
• Use graph convolutional networks (GCNs) to parametrize robot policies, enabling local information aggregation from K-hop neighbors via learnable filters.
• Define the graph using Euclidean distance-based connectivity between robots, with each robot observing its nearest neighbors and goals within a sensing radius.
• Train policies on small-scale robot configurations and transfer them zero-shot to larger swarms, leveraging the invariance of filter size to graph size.
• Employ a centralized critic for policy gradient updates during training, while inference remains fully decentralized.
• Use a sparse reward function that assigns zero reward only when all goals are covered, encouraging coverage without explicit task labeling.

Experimental results

Research questions

• RQ1 Can graph convolutional networks enable scalable, decentralized policy learning for large-scale unlabeled motion planning with constraints?
• RQ2 To what extent can policies trained on small robot swarms be transferred zero-shot to larger swarms without retraining?
• RQ3 How close is the performance of the decentralized GPG framework to a centralized, optimal solution in terms of time-to-goal?
• RQ4 Does the use of local, graph-based information aggregation outperform fully connected networks in scalability and coordination?
• RQ5 Can the framework maintain consistent performance margins relative to an optimal centralized solution as swarm size and goal complexity increase?

Key findings

• GPG achieves near-optimal performance, with time-to-goal within approximately 12–15 seconds of the centralized CAPT oracle across all tested formations.
• The performance gap between GPG and the centralized oracle remains nearly constant (~12–15 seconds) even as the number of robots increases or goal distances grow.
• Zero-shot policy transfer from small to large swarms is effective, with policies trained on smaller graphs successfully applied to larger ones without retraining.
• The framework successfully coordinates robot swarms to cover all goals without collisions, even when robots have overlapping local observations and limited sensing ranges.
• GPG demonstrates scalability by maintaining consistent neighbor interaction cardinality (fixed number of K-hop neighbors) regardless of swarm size, mitigating the curse of dimensionality.
• The use of GCNs enables local feature learning and information propagation across the robot graph, resulting in coordinated, collision-free coverage behavior.