[论文解读] PolicyCleanse: Backdoor Detection and Mitigation in Reinforcement Learning

Junfeng Guo, Ang Li+1 more|arXiv (Cornell University)|Feb 8, 2022

Adversarial Robustness in Machine Learning被引用 4

一句话总结

PolicyCleanse 提出了一种新颖的多智能体竞争强化学习后门检测与缓解框架，通过在对抗性动作序列下观察奖励衰减来识别后门触发器。该方法在检测准确率方面表现优异，并在基线方法基础上将胜率提升3%以上。

ABSTRACT

While real-world applications of reinforcement learning are becoming popular, the security and robustness of RL systems are worthy of more attention and exploration. In particular, recent works have revealed that, in a multi-agent RL environment, backdoor trigger actions can be injected into a victim agent (a.k.a. Trojan agent), which can result in a catastrophic failure as soon as it sees the backdoor trigger action. To ensure the security of RL agents against malicious backdoors, in this work, we propose the problem of Backdoor Detection in a multi-agent competitive reinforcement learning system, with the objective of detecting Trojan agents as well as the corresponding potential trigger actions, and further trying to mitigate their Trojan behavior. In order to solve this problem, we propose PolicyCleanse that is based on the property that the activated Trojan agents accumulated rewards degrade noticeably after several timesteps. Along with PolicyCleanse, we also design a machine unlearning-based approach that can effectively mitigate the detected backdoor. Extensive experiments demonstrate that the proposed methods can accurately detect Trojan agents, and outperform existing backdoor mitigation baseline approaches by at least 3% in winning rate across various types of agents and environments.

研究动机与目标

• 为应对多智能体竞争强化学习（CRL）中日益增长的后门攻击安全风险，攻击者通过对手动作注入触发器。
• 检测在暴露于攻击者特定动作序列时会灾难性失效的后门智能体。
• 通过基于机器遗忘的再训练方法缓解检测到的后门。
• 克服CRL环境中未知、可变长度连续动作触发器的挑战。
• 确保对自适应攻击的鲁棒性，此类攻击试图通过减缓失败速度来规避检测。

提出的方法

• PolicyCleanse 通过优化反向奖励策略来检测导致受害智能体奖励快速衰减的动作序列，从而实现后门检测。
• 采用期望累积奖励的二分查找方法，定位触发动作序列，通过迭代缩小阈值范围。
• 采用并行化策略优化与随机初始化，以提高在多样化环境中的检测成功率。
• 一旦识别出伪触发器，便在良性轨迹与伪触发轨迹的混合数据集上对受害策略进行再训练，以实现对后门行为的遗忘。
• 应用基于机器遗忘的缓解策略，其胜率表现优于现有基线方法。
• 在攻击者通过修改失败机制以规避检测的自适应攻击下评估该方法，模拟真实世界中的规避尝试。

实验结果

研究问题

• RQ1 能否通过观察受害智能体在遭遇特定对手动作时出现的异常奖励衰减，检测竞争性RL中的后门触发器？
• RQ2 PolicyCleanse 在多样化CRL环境与智能体架构中的后门检测效果如何？
• RQ3 所提出的缓解策略在多大程度上恢复了受害智能体的性能，相较于现有基线方法？
• RQ4 PolicyCleanse 在自适应攻击下表现如何，此类攻击通过减缓失败响应速度以规避检测？
• RQ5 检测机制能否区分自然智能体行为与人工后门触发器？

主要发现

• PolicyCleanse 在四个不同竞争环境及多种智能体架构中均以高精度成功检测出所有后门智能体。
• 该方法在所有评估环境中胜率较现有后门缓解基线方法至少提升3%。
• 通过减缓失败速度以规避检测的自适应攻击，其攻击有效性相比标准 BackdooRL 至少降低19.6%。
• 检测到的伪触发器与良性动作明显不同，且并非自然智能体行为的产物，经消融实验与可视化验证确认。
• 基于遗忘的缓解策略有效恢复了智能体性能，使受害智能体重新获得强大的竞争能力。
• 即使自适应攻击成功绕过检测，其导致的失败速度也显著减慢，降低了隐蔽性与现实世界可行性。