[Paper Review] SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving

Ming Zhou, Jun Luo+35 more|arXiv (Cornell University)|Oct 19, 2020

Reinforcement Learning in Robotics49 references104 citations

TL;DR

SMARTS is an open-source platform that enables scalable multi-agent reinforcement learning (MARL) for autonomous driving by simulating realistic, diverse agent interactions and providing benchmarks, scenarios, and evaluation metrics.

ABSTRACT

Multi-agent interaction is a fundamental aspect of autonomous driving in the real world. Despite more than a decade of research and development, the problem of how to competently interact with diverse road users in diverse scenarios remains largely unsolved. Learning methods have much to offer towards solving this problem. But they require a realistic multi-agent simulator that generates diverse and competent driving interactions. To meet this need, we develop a dedicated simulation platform called SMARTS (Scalable Multi-Agent RL Training School). SMARTS supports the training, accumulation, and use of diverse behavior models of road users. These are in turn used to create increasingly more realistic and diverse interactions that enable deeper and broader research on multi-agent interaction. In this paper, we describe the design goals of SMARTS, explain its basic architecture and its key features, and illustrate its use through concrete multi-agent experiments on interactive scenarios. We open-source the SMARTS platform and the associated benchmark tasks and evaluation metrics to encourage and empower research on multi-agent learning for autonomous driving. Our code is available at https://github.com/huawei-noah/SMARTS.

Motivation & Objective

• Motivate the need for realistic, diverse multi-agent interaction in autonomous driving (AD).
• Describe SMARTS design goals and architecture to support scalable MARL research in AD.
• Introduce bootstrapping of social agents and a modular, extensible simulation framework.
• Provide MARL-friendly observation, action, and reward interfaces and benchmark suites.
• Demonstrate MARL experiment results across progressively challenging adaptive driving scenarios.

Proposed method

• Propose a provider-based, compositional architecture with bubbles to manage interaction regions and scale compute across social agents.
• Use a domain-specific language (DSL) to compose scenarios with maps, routes, vehicles, and flows, and bubbles to hand over control to Social Agent Zoo agents.
• Integrate with distributed training frameworks (Ray, RLlib) and MARL libraries (PyMARL, MAlib), supporting CTDE and networked agent learning.
• Support heterogeneous agents (ego and social) with flexible observation and action spaces, various controllers, and configurable reward signals.
• Offer a benchmarking suite with AD-specific metrics that capture performance, behavior, and game-theoretic properties.

Experimental results

Research questions

• RQ1 How can MARL be effectively scaled to realistic, city-scale traffic interactions for autonomous driving?
• RQ2 What combination of social agents and learning strategies yields robust, safe, and efficient multi-agent driving behaviors across diverse scenarios?
• RQ3 How do different MARL paradigms (CTDE, networked learning) compare in complex driving interactions?
• RQ4 Can SMARTS provide a reproducible, extensible benchmark suite with meaningful, AD-relevant evaluation metrics?
• RQ5 What is the impact of scenario composition and bubble-based handovers on training efficiency and policy performance?

Key findings

• SMARTS enables scalable, multi-agent driving simulations with diverse, realistic social agents and modular providers.
• A broad set of MARL algorithms (including CTDE and networked variants) can be evaluated within SMARTS using the benchmark suite and AD-specific metrics.
• Experiments across two-way traffic, double merge, and unprotected intersection show varied performance and behavior across baselines, highlighting the benefits of inter-agent information sharing in complex tasks (e.g., MADDPG performing well in Intersection).
• The bubble mechanism and Social Agent Zoo support scalable, targeted training and evaluation by localizing interaction regions and distributing compute.
• The platform integrates with RLlib and MARL libraries, offering OpenAI Gym-like APIs, visualization, and a comprehensive set of observation, action, and reward interfaces.