[论文解读] Towards Safe Reinforcement Learning with a Safety Editor Policy

Haonan Yu, Wei Xu+1 more|arXiv (Cornell University)|Jan 28, 2022

Adversarial Robustness in Machine Learning被引用 12

一句话总结

SEditor 提出了一种两策略框架用于安全强化学习，其中效用最大化策略提出动作，而安全编辑策略对这些动作进行修正以确保约束违规最小化。该方法在14个安全强化学习基准测试中实现了最先进性能，约束违规率低至每2,000步一次，显著优于基线方法。

ABSTRACT

We consider the safe reinforcement learning (RL) problem of maximizing utility with extremely low constraint violation rates. Assuming no prior knowledge or pre-training of the environment safety model given a task, an agent has to learn, via exploration, which states and actions are safe. A popular approach in this line of research is to combine a model-free RL algorithm with the Lagrangian method to adjust the weight of the constraint reward relative to the utility reward dynamically. It relies on a single policy to handle the conflict between utility and constraint rewards, which is often challenging. We present SEditor, a two-policy approach that learns a safety editor policy transforming potentially unsafe actions proposed by a utility maximizer policy into safe ones. The safety editor is trained to maximize the constraint reward while minimizing a hinge loss of the utility state-action values before and after an action is edited. SEditor extends existing safety layer designs that assume simplified safety models, to general safe RL scenarios where the safety model can in theory be arbitrarily complex. As a first-order method, it is easy to implement and efficient for both inference and training. On 12 Safety Gym tasks and 2 safe racing tasks, SEditor obtains much a higher overall safety-weighted-utility (SWU) score than the baselines, and demonstrates outstanding utility performance with constraint violation rates as low as once per 2k time steps, even in obstacle-dense environments. On some tasks, this low violation rate is up to 200 times lower than that of an unconstrained RL method with similar utility performance. Code is available at https://github.com/hnyu/seditor.

研究动机与目标

• 解决在缺乏先验安全知识或预训练的环境中安全强化学习的问题。
• 将约束违规率降低至极低水平——每2,000个时间步仅发生一次，且不依赖于已知的安全模型。
• 通过双策略架构将效用最大化与安全执行解耦，实现高效、协作式学习。
• 开发一种一阶、计算高效的算法，使其能超越简化安全模型的局限并具备泛化能力。
• 实现强化学习智能体在严格安全约束下的稳健仿真到现实迁移与真实世界部署。

提出的方法

• 使用随机策略梯度下降训练效用最大化（UM）策略，以最大化效用奖励。
• 训练安全编辑（SE）策略，使其在动作编辑前后最小化效用状态-动作值之间的铰链损失，同时最大化约束奖励。
• SE策略基于当前状态s和UM提出的动作â，输出动作调整Δa，最终动作为a = h(s, â, Δa)。
• 该框架采用离策略训练与经验回放机制，以提升样本效率并实现稳定优化。
• 该方法采用原始-对偶优化方案，动态平衡效用与约束目标。
• 两个策略通过随机梯度下降联合训练，实现端到端学习，无需硬性切换或复杂启发式规则。

实验结果

研究问题

• RQ1 两策略框架是否能在安全与效用权衡方面优于单策略安全强化学习方法？
• RQ2 安全编辑策略是否能在不了解环境安全模型的前提下有效降低约束违规？
• RQ3 所提方法在高障碍物密度的多样化、复杂安全强化学习环境中表现如何？
• RQ4 该方法在缺乏先验安全信息的真实世界部署场景中，泛化能力如何？
• RQ5 该方法是否能在保持高效益性能的同时实现接近零的约束违规率？

主要发现

• 在12个Safety Gym任务和2个安全竞速任务中，SEditor在安全加权效用（SWU）得分上显著优于四个基线方法。
• 即使在障碍物高度密集的环境中，该方法的约束违规率也低至每2,000个时间步一次。
• 在某些任务中，违规率比具有相当效用性能的无约束强化学习方法低达200倍。
• 安全编辑策略在确保安全的同时有效最小化了效用损失，表现为动作编辑中铰链损失极低。
• 该方法在具有随机布局的多样化环境中表现出强大泛化能力，表明其具备稳健的安全泛化性能。
• SEditor计算效率高且可扩展，利用一阶优化和离策略训练实现快速收敛。