[Paper Review] Multi-Agent MDP Homomorphic Networks

Elise van der Pol, Herke van Hoof+2 more|arXiv (Cornell University)|Oct 9, 2021

Reinforcement Learning in Robotics36 references4 citations

TL;DR

This paper proposes Multi-Agent MDP Homomorphic Networks, a distributed deep reinforcement learning framework that enforces global symmetries—such as rotational and permutation symmetries—in cooperative multi-agent systems using a factorization of global symmetries into local transformations. The method enables data-efficient policy learning by sharing weights across symmetric state-action configurations while maintaining fully distributed execution, outperforming non-equivariant baselines in predator-prey and traffic light control tasks.

ABSTRACT

This paper introduces Multi-Agent MDP Homomorphic Networks, a class of networks that allows distributed execution using only local information, yet is able to share experience between global symmetries in the joint state-action space of cooperative multi-agent systems. In cooperative multi-agent systems, complex symmetries arise between different configurations of the agents and their local observations. For example, consider a group of agents navigating: rotating the state globally results in a permutation of the optimal joint policy. Existing work on symmetries in single agent reinforcement learning can only be generalized to the fully centralized setting, because such approaches rely on the global symmetry in the full state-action spaces, and these can result in correspondences across agents. To encode such symmetries while still allowing distributed execution we propose a factorization that decomposes global symmetries into local transformations. Our proposed factorization allows for distributing the computation that enforces global symmetries over local agents and local interactions. We introduce a multi-agent equivariant policy network based on this factorization. We show empirically on symmetric multi-agent problems that globally symmetric distributable policies improve data efficiency compared to non-equivariant baselines.

Motivation & Objective

• To address the challenge of enforcing global symmetries in cooperative multi-agent reinforcement learning while preserving distributed execution.
• To enable data-efficient learning by exploiting symmetries in the joint state-action space without requiring centralized computation.
• To extend single-agent equivariant learning techniques to multi-agent systems with complex, non-permutation symmetries such as rotations.
• To develop a factorization of global symmetries into local transformations that can be computed and enforced independently by each agent.
• To demonstrate that global equivariance improves sample efficiency in symmetric multi-agent environments.

Proposed method

• Proposes a factorization of global symmetries in the joint state-action space into local transformations acting on individual agents’ observations and actions.
• Designs a multi-agent equivariant policy network that uses regular group representations (group channels) to encode symmetry-invariant features across agents.
• Enforces equivariance by applying symmetry-preserving weight sharing across symmetric state-action pairs, using group convolution-like operations.
• Uses local communication and local computation to enforce global symmetry constraints, avoiding centralized coordination.
• Employs a message-passing or graph-based architecture to aggregate local information while preserving equivariance under global symmetry groups.
• Applies the framework to discrete-action, cooperative multi-agent environments with known symmetry groups (e.g., dihedral groups for rotation).

Experimental results

Research questions

• RQ1 Can global symmetries in cooperative multi-agent systems be exploited for data-efficient learning while maintaining distributed execution?
• RQ2 How can global symmetries—beyond simple permutations—be factorized into local, agent-level transformations for distributed policy learning?
• RQ3 Does enforcing global equivariance improve sample efficiency in symmetric multi-agent environments compared to non-equivariant baselines?
• RQ4 Can the proposed method generalize to complex coordination problems such as traffic light control with non-trivial symmetries?
• RQ5 What is the impact of symmetry enforcement on convergence speed and final policy performance in distributed settings?

Key findings

• In the predator-prey variant, the proposed method achieved effective joint policies in significantly fewer environment interactions than non-equivariant baselines.
• The method demonstrated improved data efficiency on a symmetric multi-agent coordination task, learning effective policies faster than non-equivariant counterparts.
• In the traffic light control task, enforcing global symmetries led to faster convergence and lower average vehicle wait times compared to standard MPN and data-augmented baselines.
• The equivariant policy network achieved better performance with fewer environment interactions, indicating enhanced sample efficiency.
• Even in complex coordination problems with high-dimensional state spaces, global symmetry enforcement improved learning dynamics and final policy quality.
• The approach outperformed non-equivariant distributed networks in both toy and real-world-inspired symmetric environments.