[Paper Review] Safe Deep Reinforcement Learning for Multi-Agent Systems with Continuous Action Spaces

Ziyad Sheebaelhamd, Konstantinos Zisis+4 more|arXiv (Cornell University)|Aug 9, 2021

Reinforcement Learning in Robotics19 references10 citations

TL;DR

This paper extends Safe DDPG to multi-agent deep reinforcement learning with continuous action spaces by introducing a soft-constrained safety layer to the MADDPG framework. By reformulating the safety projection as a penalized optimization problem, the method ensures recursive feasibility and reduces constraint violations by 97.7% under unsafe conditions, enabling safe training without backup policies.

ABSTRACT

Multi-agent control problems constitute an interesting area of application for deep reinforcement learning models with continuous action spaces. Such real-world applications, however, typically come with critical safety constraints that must not be violated. In order to ensure safety, we enhance the well-known multi-agent deep deterministic policy gradient (MADDPG) framework by adding a safety layer to the deep policy network. In particular, we extend the idea of linearizing the single-step transition dynamics, as was done for single-agent systems in Safe DDPG (Dalal et al., 2018), to multi-agent settings. We additionally propose to circumvent infeasibility problems in the action correction step using soft constraints (Kerrigan & Maciejowski, 2000). Results from the theory of exact penalty functions can be used to guarantee constraint satisfaction of the soft constraints under mild assumptions. We empirically find that the soft formulation achieves a dramatic decrease in constraint violations, making safety available even during the learning procedure.

Motivation & Objective

• To address safety constraints in multi-agent deep reinforcement learning with continuous action spaces, where hard constraints may lead to infeasible action corrections.
• To extend Safe DDPG to multi-agent settings while overcoming the limitation of recursive infeasibility in hard-constrained optimization.
• To improve robustness during training under unsafe initialization and exogenous disturbances by ensuring the safety filter remains active.
• To maintain effective learning performance while minimizing constraint violations through a soft-constrained formulation.
• To eliminate reliance on backup controllers by enabling the safety filter to operate even when constraints are tight or conflicting.

Proposed method

• Integrates a safety layer into the MADDPG framework that projects actions onto a safe set using a quadratic program based on linearized constraint functions.
• Applies a first-order Taylor approximation of constraints in the action space, parameterized by a neural network trained on historical data.
• Reformulates the hard-constrained optimization as a soft-constrained problem using exact penalty functions to allow controlled constraint violations with penalty.
• Employs soft constraints via a tolerance margin, as proposed by Kerrigan & Maciejowski (2000), to ensure recursive feasibility.
• Uses theory of exact penalty functions to guarantee constraint satisfaction under mild assumptions when the penalty is appropriately tuned.
• Maintains end-to-end differentiability and compatibility with off-policy deep RL training by embedding the safety layer within the policy network.

Experimental results

Research questions

• RQ1 Can the Safe DDPG framework be successfully extended to multi-agent systems with continuous action spaces?
• RQ2 Does the use of soft constraints improve recursive feasibility and reduce constraint violations compared to hard constraints in multi-agent settings?
• RQ3 How does the soft-constrained safety layer affect learning performance and convergence in the presence of unsafe initialization or disturbances?
• RQ4 Can the safety filter remain active during training without requiring a separate backup controller?
• RQ5 To what extent does the soft-constrained formulation maintain task performance while ensuring safety?

Key findings

• Under unsafe initialization (UI), soft MADDPG reduced collisions by 97.71% compared to unconstrained MADDPG, while hard MADDPG only achieved a 84.38% reduction.
• Under exogenous disturbances (ED), soft MADDPG reduced collisions by 97.99%, compared to 42.42% for hard MADDPG.
• The hard-constrained formulation suffered from infeasibility in 56.7% of episodes under ED conditions and 20.9% under UI, limiting its practical use.
• In test simulations, soft MADDPG achieved the lowest cumulative collisions in both UI and ED settings, demonstrating superior safety robustness.
• The soft-constrained approach maintained similar learning trends and final returns to unconstrained MADDPG, indicating no negative impact on task performance.
• Constraint violations were dramatically reduced during training, with soft MADDPG achieving only 2.0% of the baseline collision count in the ED case.