[Paper Review] PettingZoo: Gym for Multi-Agent Reinforcement Learning

Justin K. Terry, Benjamin Black+11 more|arXiv (Cornell University)|Sep 30, 2020

Reinforcement Learning in Robotics35 references146 citations

TL;DR

PettingZoo introduces a universal Python API and the Agent Environment Cycle (AEC) model to standardize and accelerate MARL research, addressing limitations of POSG/EFG-based APIs.

ABSTRACT

This paper introduces the PettingZoo library and the accompanying Agent Environment Cycle ("AEC") games model. PettingZoo is a library of diverse sets of multi-agent environments with a universal, elegant Python API. PettingZoo was developed with the goal of accelerating research in Multi-Agent Reinforcement Learning ("MARL"), by making work more interchangeable, accessible and reproducible akin to what OpenAI's Gym library did for single-agent reinforcement learning. PettingZoo's API, while inheriting many features of Gym, is unique amongst MARL APIs in that it's based around the novel AEC games model. We argue, in part through case studies on major problems in popular MARL environments, that the popular game models are poor conceptual models of games commonly used in MARL and accordingly can promote confusing bugs that are hard to detect, and that the AEC games model addresses these problems.

Motivation & Objective

• Propose a standard, Gym-like API for MARL to improve accessibility, interchangeability, and reproducibility.
• Introduce the Agent Environment Cycle (AEC) game model as a better conceptual fit for code-based MARL environments.
• Demonstrate how AEC resolves conceptual and practical bugs seen in POSG/EFG-based MARL implementations.
• Provide a library of diverse default environments and show adoption across libraries and courses.

Proposed method

• Define the AEC games model and prove its equivalence with POSG in both directions (AEC ⇔ POSG).
• Design the PettingZoo API to resemble Gym while supporting many agents, agent death/creation, and dynamic participation.
• Introduce agent_iter and last methods to manage sequential agent actions and observed rewards.
• Provide a universal API with agent-keyed spaces and low-level access for experimental flexibility.
• Bundle 63 default environments and ensure compatibility with major MARL libraries for broad adoption.

Experimental results

Research questions

• RQ1 Can an API based on the AEC model provide a more intuitive and robust foundation for MARL than POSG/EFG-based APIs?
• RQ2 Does the PettingZoo API support varied agent counts, agent creation/death, and dynamic participation without introducing race conditions?
• RQ3 How well does the AEC-based API integrate with existing MARL libraries and environments, and what is the practical impact on reproducibility and bugs?
• RQ4 What is the relationship between AEC games and POSG in terms of representational capability?

Key findings

• AEC games provide clearer reward attribution and reduce race-condition bugs compared to POSG-based APIs.
• Every AEC game has an equivalent POSG and vice versa, showing AEC’s theoretical completeness relative to POSG.
• PettingZoo offers a Gym-like Python API with agent_iter and last to handle sequential agent actions and dynamic agent sets.
• The library hosts 63 default environments and integrates with major MARL libraries, aiding adoption and education.
• Case studies illustrate POSG-induced bugs in existing environments that AEC-based APIs can prevent.