[Paper Review] Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificates

Zengyi Qin, Kaiqing Zhang+3 more|arXiv (Cornell University)|Jan 14, 2021

Advanced Control Systems Optimization36 references48 citations

TL;DR

The paper presents a decentralized, joint-learning framework that trains multi-agent control policies with neural control barrier certificates (CBFs) to ensure safety and scalability across many agents.

ABSTRACT

We study the multi-agent safe control problem where agents should avoid collisions to static obstacles and collisions with each other while reaching their goals. Our core idea is to learn the multi-agent control policy jointly with learning the control barrier functions as safety certificates. We propose a novel joint-learning framework that can be implemented in a decentralized fashion, with generalization guarantees for certain function classes. Such a decentralized framework can adapt to an arbitrarily large number of agents. Building upon this framework, we further improve the scalability by incorporating neural network architectures that are invariant to the quantity and permutation of neighboring agents. In addition, we propose a new spontaneous policy refinement method to further enforce the certificate condition during testing. We provide extensive experiments to demonstrate that our method significantly outperforms other leading multi-agent control approaches in terms of maintaining safety and completing original tasks. Our approach also shows exceptional generalization capability in that the control policy can be trained with 8 agents in one scenario, while being used on other scenarios with up to 1024 agents in complex multi-agent environments and dynamics.

Motivation & Objective

• Motivate safe control for large-scale multi-agent systems with formal safety guarantees.
• Develop a decentralized framework to jointly learn control policies and safety certificates (CBFs).
• Achieve scalability to arbitrarily many agents via local observations and invariant network design.
• Demonstrate generalization to unseen environments and agent counts.
• Provide techniques to improve training efficiency and testing safety.

Proposed method

• Define decentralized CBFs that depend on local state and observations.
• Jointly learn policies pi_i and barrier functions h_i via data with a margin gamma.
• Use a loss that enforces the three CBF conditions and a goal-reaching term.
• Adopt quantity-permutation invariant neural encoders for dynamic neighbor counts.
• Train with on-policy data and iterative data collection to align training/testing distributions.
• Introduce spontaneous online policy refinement to adjust actions when CBF conditions are violated.

Experimental results

Research questions

• RQ1 Can decentralized control barrier functions guarantee safety in multi-agent settings without a central controller?
• RQ2 How can neural networks be designed to handle variable numbers and permutations of neighboring agents?
• RQ3 Do joint-learning of policies and CBFs generalize to unseen scenarios and larger agent counts?
• RQ4 Does online refinement improve safety during testing beyond learned certificates?
• RQ5 What are practical training strategies to ensure safety while achieving goal-directed behavior?

Key findings

• The joint-learning framework with decentralized CBFs yields safety guarantees when the decentralized conditions are satisfied.
• Using quantity-permutation invariant encoders allows the method to scale to arbitrarily many agents and varying neighbor counts.
• Policies trained on 8 agents generalize to scenarios with up to 1024 agents in complex environments.
• The method outperforms leading learning-based and planning approaches in safety and task completion across 2D and 3D tasks.
• Spontaneous online policy refinement further improves safety during testing by actively enforcing CBF conditions.
• Experiments demonstrate strong generalization across environments and agent counts beyond training conditions.