[Paper Review] Adversarial Reinforcement Learning under Partial Observability in Autonomous Computer Network Defence

Yi Han, David Hubczenko+7 more|arXiv (Cornell University)|Feb 25, 2019

Adversarial Robustness in Machine Learning51 references4 citations

TL;DR

This paper proposes an inversion-based defense mechanism to counter causative adversarial attacks in reinforcement learning (RL) agents used for autonomous cyber defense under partial observability. Even with limited visibility into the network, attackers can poison RL training via state perturbations; the proposed method reverses these perturbations, significantly reducing attack impact without degrading performance in non-attack scenarios.

ABSTRACT

Recent studies have demonstrated that reinforcement learning (RL) agents are susceptible to adversarial manipulation, similar to vulnerabilities previously demonstrated in the supervised learning setting. While most existing work studies the problem in the context of computer vision or console games, this paper focuses on reinforcement learning in autonomous cyber defence under partial observability. We demonstrate that under the black-box setting, where the attacker has no direct access to the target RL model, causative attacks---attacks that target the training process---can poison RL agents even if the attacker only has partial observability of the environment. In addition, we propose an inversion defence method that aims to apply the opposite perturbation to that which an attacker might use to generate their adversarial samples. Our experimental results illustrate that the countermeasure can effectively reduce the impact of the causative attack, while not significantly affecting the training process in non-attack scenarios.

Motivation & Objective

• Address the vulnerability of RL agents in autonomous cyber defense to causative attacks under partial observability, where attackers have limited network visibility.
• Overcome the limitations of prior work that assumed full observability for both attacker and defender, which is unrealistic in real-world networks.
• Develop a defense mechanism effective in partial observability settings where existing methods fail, without requiring prior knowledge of attack patterns.
• Ensure the defense mechanism maintains high performance in non-attacked training scenarios to avoid introducing new vulnerabilities.

Proposed method

• Model the network defense problem as a Markov Decision Process (MDP) with a DQN-based RL agent trained to protect a critical server and maintain network connectivity.
• Introduce a causative attack model where the attacker perturbs observed states during training to mislead the RL agent, even with partial knowledge of the network topology.
• Design an inversion-based defense that detects and reverses adversarial perturbations by applying the opposite transformation to the state input before training.
• The defense operates without prior knowledge of the attacker’s perturbation strategy, relying on statistical or structural inversion to neutralize poisoning effects.
• Use a black-box attack setting where the attacker cannot access the internal model parameters, simulating real-world constraints.
• Evaluate the defense using a realistic SDN-based network with 100 nodes and 172 links, under two attacker observability levels (1/3 and 1/2 of nodes).

Experimental results

Research questions

• RQ1 Can causative attacks effectively poison RL training in autonomous cyber defense when the attacker has only partial observability of the network?
• RQ2 How does partial observability affect the attacker’s ability to manipulate the RL agent’s policy through state perturbations?
• RQ3 Can an inversion-based defense mechanism effectively counteract causative attacks without prior knowledge of the attack pattern?
• RQ4 What is the performance impact of the defense mechanism in non-attack scenarios, and does it maintain the RL agent’s learning efficiency?

Key findings

• Causative attacks that perturb training states can successfully mislead RL agents into suboptimal policies, even when the attacker has only partial observability of the network.
• The proposed inversion defense significantly reduces the success rate of causative attacks, restoring the RL agent’s ability to learn optimal defense strategies.
• The defense mechanism has minimal negative impact on training performance in non-attacked scenarios, preserving the agent’s learning efficiency.
• The method remains effective even when the attacker observes only 34 out of 100 nodes (1/3), demonstrating robustness under realistic partial observability conditions.
• The inversion-based approach outperforms prior defense mechanisms in the new partial observability setting, where earlier methods failed.
• The results confirm that adversarial training and other existing defenses are insufficient in this realistic threat model, highlighting the need for new defense strategies.