[论文解读] Characterizing Attacks on Deep Reinforcement Learning

Xinlei Pan, Chaowei Xiao+9 more|arXiv (Cornell University)|Jul 21, 2019

Adversarial Robustness in Machine Learning参考文献 32被引用 52

一句话总结

这篇论文开发了现实、高效的黑箱和在线对深度强化学习系统的对抗攻击，包括观测和环境动力学，并在仿真和真实机器人上进行了验证。

ABSTRACT

Recent studies show that Deep Reinforcement Learning (DRL) models are vulnerable to adversarial attacks, which attack DRL models by adding small perturbations to the observations. However, some attacks assume full availability of the victim model, and some require a huge amount of computation, making them less feasible for real world applications. In this work, we make further explorations of the vulnerabilities of DRL by studying other aspects of attacks on DRL using realistic and efficient attacks. First, we adapt and propose efficient black-box attacks when we do not have access to DRL model parameters. Second, to address the high computational demands of existing attacks, we introduce efficient online sequential attacks that exploit temporal consistency across consecutive steps. Third, we explore the possibility of an attacker perturbing other aspects in the DRL setting, such as the environment dynamics. Finally, to account for imperfections in how an attacker would inject perturbations in the physical world, we devise a method for generating a robust physical perturbations to be printed. The attack is evaluated on a real-world robot under various conditions. We conduct extensive experiments both in simulation such as Atari games, robotics and autonomous driving, and on real-world robotics, to compare the effectiveness of the proposed attacks with baseline approaches. To the best of our knowledge, we are the first to apply adversarial attacks on DRL systems to physical robots.

研究动机与目标

• 在现实攻击情境下激发和理解DRL的脆弱性。
• 开发不需要访问受害模型参数的高效黑箱对抗攻击。
• 利用MDP的时间结构来创建高吞吐量的在线攻击。
• 探索对环境动力学的扰动以及攻击在现实世界的鲁棒性。
• 证明在真实机器人上进行鲁棒物理扰动的对抗攻击的可行性。

提出的方法

• 采用并改进基于有限差分的黑箱攻击，结合自适应采样以降低梯度估计成本。
• 提出利用时间一致性的在线顺序攻击，通过单次扰动攻击多帧。
• 引入帧选择策略以识别对攻击生成重要的帧。
• 开发针对环境转移动力学的攻击，方法包括随机搜索和基于强化学习的动力学搜索。
• 通过生成对现实世界条件鲁棒的印刷对抗贴片并在实际条件下评估，将攻击扩展到物理机器人。

实验结果

研究问题

• RQ1 黑箱攻击在DRL设置中能否达到或超过白箱/黑箱基线？
• RQ2 自适应采样（SFD）是否比标准有限差分提高梯度估计效率？
• RQ3 在线顺序攻击是否在吞吐量和效能上优于逐帧攻击？
• RQ4 对环境动力学的扰动是否对DRL代理可行且有效？
• RQ5 对抗贴片在物理机器人部署中是否鲁棒？

主要发现

• 使用自适应采样和有限差分的黑箱攻击在无需模型结构或参数访问的情况下也能有效。
• 通过扰动少量帧并将扰动应用于后续帧，在线顺序攻击实现高吞吐量。
• 通过随机搜索和基于强化学习的方法对环境动力学的攻击可以使代理性能下降到超出基线水平。
• 可打造对打印和视角变化鲁棒的物理对抗贴片，在机器人导航任务中表现鲁棒。
• 实验覆盖Atari游戏、MuJoCo控制任务、TORCS驾驶仿真和真实机器人实验，並与白箱和基线黑箱方法比较。