[论文解读] OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research

Jiaming Ji, Jiayi Zhou+8 more|arXiv (Cornell University)|May 16, 2023

Adversarial Robustness in Machine Learning被引用 12

一句话总结

OmniSafe 提供一个模块化的高性能开源框架，通过提供广泛的安全强化学习算法、分布式训练和详尽的文档来加速 SafeRL 研究。

ABSTRACT

AI systems empowered by reinforcement learning (RL) algorithms harbor the immense potential to catalyze societal advancement, yet their deployment is often impeded by significant safety concerns. Particularly in safety-critical applications, researchers have raised concerns about unintended harms or unsafe behaviors of unaligned RL agents. The philosophy of safe reinforcement learning (SafeRL) is to align RL agents with harmless intentions and safe behavioral patterns. In SafeRL, agents learn to develop optimal policies by receiving feedback from the environment, while also fulfilling the requirement of minimizing the risk of unintended harm or unsafe behavior. However, due to the intricate nature of SafeRL algorithm implementation, combining methodologies across various domains presents a formidable challenge. This had led to an absence of a cohesive and efficacious learning framework within the contemporary SafeRL research milieu. In this work, we introduce a foundational framework designed to expedite SafeRL research endeavors. Our comprehensive framework encompasses an array of algorithms spanning different RL domains and places heavy emphasis on safety elements. Our efforts are to make the SafeRL-related research process more streamlined and efficient, therefore facilitating further research in AI safety. Our project is released at: https://github.com/PKU-Alignment/omnisafe.

研究动机与目标

• 激发在 SafeRL 中需要一个统一的开源软件框架的需求，以解决 RL 部署中的安全性问题。
• 将 OmniSafe 呈现为一个模块化基础设施，支持 On-Policy、Off-Policy、基于模型和离线等不同类别的多样 SafeRL 算法。
• 展示高性能并行计算以及环境/代理并行性，以加速 SafeRL 实验。
• 通过广泛的测试和文档确保代码的可靠性、可重复性和社区的增长。
• 概述面向标准化 SafeRL 研究工具与方法的未来方向。

提出的方法

• 通过使用 Adapter 和 Wrapper 组件对算法层级进行模块化抽象来处理 CMDP 和环境差异，引入 OmniSafe。
• 利用 torch.distributed 实现环境级异步并行和代理异步学习，以加速训练。
• 在 Safety-Gym 和 Mujoco-Velocity 环境中提供广泛测试以验证算法实现。
• 提供全面的 API 文档、教程和开发者指南，以促进采用和可重复性。
• 描述一个统一的数据流框架，在该框架中轨迹被生成、预处理、学习并通过循环数据流过程转化为动作。

Figure 1 : The core features of OmniSafe include (a) Comprehensive API documentation with user guides, examples, and best practices for efficient learning, the documentation can be found in https://omnisafe.readthedocs.io ; (b) Streamlined algorithm training through single-file execution, simplifyin

实验结果

研究问题

• RQ1 在统一的 OSS 框架中需要支持哪些 SafeRL 算法和范式？
• RQ2 模块化的适配器和包装器如何协调跨领域的 CMDP 和环境差异，以简化 SafeRL 研究？
• RQ3 分布式与异步学习对 SafeRL 训练速度和稳定性有何影响？
• RQ4 OmniSafe 如何确保 SafeRL 实验的可靠性和可重复性？
• RQ5 OmniSafe 如何促进 SafeRL 工具的社区增长和标准化？

主要发现

• OmniSafe 提供一个高度模块化的框架，覆盖 On-Policy、Off-Policy、Offline 和 Model-based 等几十种 SafeRL 算法。
• Adapter 与 Wrapper 的设计实现跨域兼容性，当整合新环境或问题范式时可降低工程量。
• 通过 torch.distributed 的分布式训练实现环境级异步并行和代理异步学习，加速训练并提升稳定性。
• 在 Safety-Gym 和 Mujoco-Velocity 环境中的广泛测试表明结果与原论文一致，支持结果可重复性。
• 全面的 API 文档、教程和开发者指南有助于 SafeRL 研究的采用、可重复性以及社区增长。

Figure 2 : A high-level depiction of OmniSafe ’s distributed dataflow process. Each process periodically syncs weights and all-reduce gradients with other processes. Vectorized Environments first generate trajectories of the agent’s interactions with the environment. Second, the EnvWrapper monitors