[Paper Review] Reinforcement Learning for Feedback-Enabled Cyber Resilience

Yunhan Huang, Linan Huang+1 more|arXiv (Cornell University)|Jul 2, 2021

Information and Cyber Security4 citations

TL;DR

This paper proposes a reinforcement learning (RL)-based feedback architecture for cyber resilience, enabling real-time adaptation to known and zero-day threats across posture, information, and human-related vulnerabilities. It demonstrates that RL can effectively drive adaptive defense mechanisms like moving target defense and honeypots, while also exposing RL's own vulnerabilities to reward, observation, and action manipulation by attackers.

ABSTRACT

Digitization and remote connectivity have enlarged the attack surface and made cyber systems more vulnerable. As attackers become increasingly sophisticated and resourceful, mere reliance on traditional cyber protection, such as intrusion detection, firewalls, and encryption, is insufficient to secure the cyber systems. Cyber resilience provides a new security paradigm that complements inadequate protection with resilience mechanisms. A Cyber-Resilient Mechanism (CRM) adapts to the known or zero-day threats and uncertainties in real-time and strategically responds to them to maintain critical functions of the cyber systems in the event of successful attacks. Feedback architectures play a pivotal role in enabling the online sensing, reasoning, and actuation process of the CRM. Reinforcement Learning (RL) is an essential tool that epitomizes the feedback architectures for cyber resilience. It allows the CRM to provide sequential responses to attacks with limited or without prior knowledge of the environment and the attacker. In this work, we review the literature on RL for cyber resilience and discuss cyber resilience against three major types of vulnerabilities, i.e., posture-related, information-related, and human-related vulnerabilities. We introduce three application domains of CRMs: moving target defense, defensive cyber deception, and assistive human security technologies. The RL algorithms also have vulnerabilities themselves. We explain the three vulnerabilities of RL and present attack models where the attacker targets the information exchanged between the environment and the agent: the rewards, the state observations, and the action commands. We show that the attacker can trick the RL agent into learning a nefarious policy with minimum attacking effort. Lastly, we discuss the future challenges of RL for cyber security and resilience and emerging applications of RL-based CRMs.

Motivation & Objective

• Address the limitations of traditional cyber protection against advanced persistent threats (APTs) and zero-day exploits.
• Develop a feedback-enabled cyber-resilient mechanism (CRM) that adapts in real time to unknown and evolving threats.
• Integrate reinforcement learning (RL) into CRM to enable strategic, sequential responses without prior environmental knowledge.
• Identify and analyze vulnerabilities in RL itself, particularly in reward, state observation, and action command channels.
• Explore emerging applications of RL-based CRMs in moving target defense, cyber deception, and assistive human security technologies.

Proposed method

• Formulate a P2R2 CRM framework with four stages: Preparation, Protection, Response, and Recovery.
• Apply model-free and value-based RL algorithms (e.g., Q-learning, deep Q-networks) to learn optimal defense policies in dynamic environments.
• Design adaptive moving target defense (MTD) strategies using RL to balance security and usability.
• Implement self-configuring honeypots via RL to optimize attacker engagement and deception effectiveness.
• Develop RL-based alert and attention management systems to counter intentional overload (e.g., IDoS attacks).
• Model adversarial attacks on RL components—reward poisoning, observation manipulation, and action command spoofing—using game-theoretic frameworks.

Experimental results

Research questions

• RQ1 How can RL be effectively integrated into feedback architectures to enable real-time, adaptive cyber resilience?
• RQ2 What are the key vulnerabilities in RL-based defense systems when exposed to adversarial manipulation of rewards, observations, or actions?
• RQ3 How does RL improve resilience against posture-related, information-related, and human-related cyber vulnerabilities?
• RQ4 In what ways can RL enhance moving target defense and defensive deception strategies in critical infrastructure?
• RQ5 What are the minimal attack efforts required to subvert RL agents through manipulation of feedback signals?

Key findings

• RL enables effective, adaptive defense in cyber resilience by learning optimal policies in real time with limited prior knowledge of the environment or attacker behavior.
• Reward poisoning attacks can successfully mislead RL agents into learning suboptimal or malicious policies with minimal intervention.
• Adversaries can manipulate state observations or action commands to induce incorrect decisions, demonstrating inherent vulnerabilities in RL-based CRMs.
• RL-based moving target defense strategies achieve improved security-usability trade-offs by dynamically reconfiguring system parameters.
• Self-adaptive honeypots configured via Q-learning significantly increase attacker engagement time and detection accuracy.
• RL-based attention management reduces alert fatigue and improves response efficiency in high-volume intrusion detection scenarios.