[论文解读] Design of intentional backdoors in sequential models

Zhaoyuan Yang, Naresh Iyer+2 more|arXiv (Cornell University)|Feb 26, 2019

Adversarial Robustness in Machine Learning参考文献 22被引用 19

一句话总结

本文提出了一种新型后门攻击方法，针对基于长短期记忆（LSTM）的强化学习智能体，通过嵌入触发器来改变其长期决策行为。该攻击通过操纵隐藏状态和细胞状态，在短暂触发暴露后永久性地改变策略行为，即使在移除触发器后仍表现出持续的误分类现象。

ABSTRACT

Recent work has demonstrated robust mechanisms by which attacks can be orchestrated on machine learning models. In contrast to adversarial examples, backdoor or trojan attacks embed surgically modified samples with targeted labels in the model training process to cause the targeted model to learn to misclassify chosen samples in the presence of specific triggers, while keeping the model performance stable across other nominal samples. However, current published research on trojan attacks mainly focuses on classification problems, which ignores sequential dependency between inputs. In this paper, we propose methods to discreetly introduce and exploit novel backdoor attacks within a sequential decision-making agent, such as a reinforcement learning agent, by training multiple benign and malicious policies within a single long short-term memory (LSTM) network. We demonstrate the effectiveness as well as the damaging impact of such attacks through initial outcomes generated from our approach, employed on grid-world environments. We also provide evidence as well as intuition on how the trojan trigger and malicious policy is activated. Challenges with network size and unintentional triggers are identified and analogies with adversarial examples are also discussed. In the end, we propose potential approaches to defend against or serve as early detection for such attacks. Results of our work can also be extended to many applications of LSTM and recurrent networks.

研究动机与目标

• 识别并展示一种新型后门攻击，针对序列模型，利用LSTM中的长期依赖特性。
• 展示如何通过短暂、精准定位的触发器永久改变智能体策略，而不会影响在正常输入上的性能。
• 分析意外触发激活的机制，并探讨其与序列上下文下对抗样本的相似性。
• 提出针对LSTM内部状态和参数分布的检测与防御机制。

提出的方法

• 训练单一LSTM网络，通过多任务学习同时学习良性策略与恶意策略。
• 在输入序列的单个帧中嵌入触发器，通过改变隐藏状态和细胞状态来激活恶意策略。
• 使用网格世界环境模拟序列决策过程，评估触发器对长期行为的影响。
• 监控LSTM内部状态（隐藏状态与细胞状态），检测后门激活的异常变化。
• 分析LSTM单元间的参数分布，识别与后门存在相关的偏差。
• 提出在线监控系统，作为类似免疫系统的机制，用于检测并重置异常内部状态。

实验结果

研究问题

• RQ1 如何设计后门，使得在极短触发暴露下即可永久改变基于LSTM的强化学习智能体的长期决策行为？
• RQ2 LSTM架构中的何种机制使得短暂触发能够引发持久的行为改变？
• RQ3 非故意触发——即常见观测序列——如何导致与故意后门相似的误分类？
• RQ4 在内部状态动态与参数分布方面，后门LSTM具有哪些可区分的特征？
• RQ5 哪些防御策略能够有效实现实时检测或中和此类后门攻击？

主要发现

• 单个包含触发器的帧即可永久改变基于LSTM的智能体策略，即使触发器已不再出现在输入序列中。
• 恶意策略通过操纵细胞状态（其用于存储长期目标）被激活，从而导致行为的持久性偏移。
• 非故意触发（常见观测序列）同样可引发长期目标的突然且有害变化，其表现类似于序列模型中的对抗样本。
• 后门智能体表现出异常的内部状态模式，尤其体现在细胞状态上，可通过监控实现检测。
• 参数分布分析表明，后门模型更倾向于使用更多LSTM细胞单元来存储信息，这可能构成一种潜在的检测指纹。
• 对LSTM内部状态的在线监控可作为实时防御机制，类似于免疫系统，用于检测并重置异常行为。