[论文解读] Reinforcement Learning for Feedback-Enabled Cyber Resilience

Yunhan Huang, Linan Huang+1 more|arXiv (Cornell University)|Jul 2, 2021

Information and Cyber Security被引用 4

一句话总结

本文提出了一种基于强化学习（RL）的反馈架构，用于网络弹性，能够在姿态、信息和人为相关漏洞方面实时适应已知和零日威胁。结果表明，RL可有效驱动自适应防御机制（如移动目标防御和蜜罐），同时揭示了RL自身在奖励、观测和动作操控方面易受攻击的漏洞。

ABSTRACT

Digitization and remote connectivity have enlarged the attack surface and made cyber systems more vulnerable. As attackers become increasingly sophisticated and resourceful, mere reliance on traditional cyber protection, such as intrusion detection, firewalls, and encryption, is insufficient to secure the cyber systems. Cyber resilience provides a new security paradigm that complements inadequate protection with resilience mechanisms. A Cyber-Resilient Mechanism (CRM) adapts to the known or zero-day threats and uncertainties in real-time and strategically responds to them to maintain critical functions of the cyber systems in the event of successful attacks. Feedback architectures play a pivotal role in enabling the online sensing, reasoning, and actuation process of the CRM. Reinforcement Learning (RL) is an essential tool that epitomizes the feedback architectures for cyber resilience. It allows the CRM to provide sequential responses to attacks with limited or without prior knowledge of the environment and the attacker. In this work, we review the literature on RL for cyber resilience and discuss cyber resilience against three major types of vulnerabilities, i.e., posture-related, information-related, and human-related vulnerabilities. We introduce three application domains of CRMs: moving target defense, defensive cyber deception, and assistive human security technologies. The RL algorithms also have vulnerabilities themselves. We explain the three vulnerabilities of RL and present attack models where the attacker targets the information exchanged between the environment and the agent: the rewards, the state observations, and the action commands. We show that the attacker can trick the RL agent into learning a nefarious policy with minimum attacking effort. Lastly, we discuss the future challenges of RL for cyber security and resilience and emerging applications of RL-based CRMs.

研究动机与目标

• 解决传统网络安全防护在应对高级持续性威胁（APTs）和零日漏洞利用方面的局限性。
• 开发一种具备反馈能力的网络弹性机制（CRM），可实时适应未知且不断演变的威胁。
• 将强化学习（RL）集成到CRM中，实现在缺乏环境先验知识情况下的战略性、序列性响应。
• 识别并分析RL自身存在的漏洞，特别是奖励、状态观测和动作命令通道的脆弱性。
• 探索基于RL的CRM在移动目标防御、网络欺骗以及辅助人类安全技术中的新兴应用。

提出的方法

• 提出P2R2 CRM框架，包含四个阶段：准备、防护、响应和恢复。
• 应用无模型且基于价值的RL算法（如Q-learning、深度Q网络）以在动态环境中学习最优防御策略。
• 利用RL设计自适应移动目标防御（MTD）策略，以在安全性和可用性之间取得平衡。
• 通过RL实现自配置蜜罐，以优化攻击者参与度和欺骗有效性。
• 开发基于RL的警报与注意力管理系统，以应对有意的过载攻击（如IDoS攻击）。
• 使用博弈论框架对RL组件的对抗性攻击进行建模，包括奖励污染、观测操纵和动作命令欺骗。

实验结果

研究问题

• RQ1 如何有效将RL集成到反馈架构中，以实现实时、自适应的网络弹性？
• RQ2 当RL防御系统暴露于对奖励、观测或动作的对抗性操纵时，其关键漏洞是什么？
• RQ3 RL在应对姿态相关、信息相关和人为相关网络漏洞方面如何提升弹性？
• RQ4 RL在关键基础设施中如何增强移动目标防御和防御欺骗策略？
• RQ5 通过操纵反馈信号，使RL智能体失效所需的最小攻击成本是多少？

主要发现

• RL通过在环境和攻击者行为先验知识有限的情况下实时学习最优策略，有效实现了自适应防御。
• 奖励污染攻击可仅通过极少干预即成功误导RL智能体，使其学习次优或恶意策略。
• 攻击者可通过操纵状态观测或动作命令，诱导RL智能体做出错误决策，表明基于RL的CRM存在固有脆弱性。
• 基于RL的移动目标防御策略通过动态重构系统参数，实现了更优的安全性与可用性权衡。
• 通过Q-learning配置的自适应蜜罐显著增加了攻击者参与时间与检测准确率。
• 基于RL的注意力管理系统可减少警报疲劳，并在高流量入侵检测场景中提升响应效率。