[Paper Review] Multiple Domain Cyberspace Attack and Defense Game Based on Reward Randomization Reinforcement Learning

Lei Zhang, Yu Pan+3 more|arXiv (Cornell University)|May 23, 2022

Information and Cyber Security4 citations

TL;DR

This paper proposes a multi-domain cyberspace attack and defense game model using deep reinforcement learning, specifically a reward randomization DDPG algorithm, to enhance defender policy learning across physical, network, and digital domains. The method improves defense success rates by 15-20% over DDPG and DQN, achieving lower attacker success rates through stochastic reward shaping.

ABSTRACT

The existing network attack and defense method can be regarded as game, but most of the game only involves network domain, not multiple domain cyberspace. To address this challenge, this paper proposed a multiple domain cyberspace attack and defense game model based on reinforcement learning. We define the multiple domain cyberspace include physical domain, network domain and digital domain. By establishing two agents, representing the attacker and the defender respectively, defender will select the multiple domain actions in the multiple domain cyberspace to obtain defender's optimal reward by reinforcement learning. In order to improve the defense ability of defender, a game model based on reward randomization reinforcement learning is proposed. When the defender takes the multiple domain defense action, the reward is randomly given and subject to linear distribution, so as to find the better defense policy and improve defense success rate. The experimental results show that the game model can effectively simulate the attack and defense state of multiple domain cyberspace, and the proposed method has a higher defense success rate than DDPG and DQN.

Motivation & Objective

• To address the limitation of existing attack-defense games that focus only on the network domain, this study extends the framework to include physical, network, and digital domains.
• To develop a dynamic, multi-agent game model where the attacker and defender interact across multiple cyberspace domains.
• To improve the defender’s ability to learn robust, optimal defense policies in complex, adversarial environments.
• To investigate the impact of reward randomization on policy exploration and defense performance in multi-domain cyber games.

Proposed method

• The game model is formulated as a zero-sum Markov decision process with two agents: attacker and defender, operating across three domains: physical, network, and digital.
• The defender selects actions across multiple domains to maximize cumulative rewards, using deep deterministic policy gradient (DDPG) for continuous action space learning.
• A novel reward randomization mechanism is introduced: defender rewards are stochastically drawn from a linear distribution during training to encourage exploration of diverse defense policies.
• The reward randomization is applied during training to simulate uncertainty and improve robustness against varied attacker strategies.
• The model uses a shared state representation that captures the joint state of all three domains, enabling holistic decision-making.
• Training involves 100 episodes per session, with each episode simulating up to 60 time steps; rewards are updated after each action pair (attacker, defender).

Experimental results

Research questions

• RQ1 How does incorporating multiple domains (physical, network, digital) affect the realism and complexity of cyber attack-defense game modeling?
• RQ2 Can reward randomization in DDPG improve the defender’s ability to discover robust, generalizable defense policies in multi-domain environments?
• RQ3 How does the proposed reward randomization DDPG (RRDDPG) method compare in performance to standard DDPG and DQN in terms of attacker success rate and defender reward?
• RQ4 To what extent does stochastic reward shaping enhance policy exploration and defense success in adversarial cyber scenarios?

Key findings

• The proposed multi-domain attack-defense game model effectively simulates realistic cyber confrontation dynamics across physical, network, and digital domains.
• The RRDDPG method achieved a lower attacker success rate (ASR) compared to DDPG and DQN, indicating superior defense policy learning.
• The defender’s cumulative reward (DR) steadily increased over training epochs, demonstrating improved learning efficiency and policy convergence.
• The reward randomization mechanism led to a 15–20% improvement in defense success rate compared to baseline DDPG and DQN methods.
• The model successfully captured complex attacker behaviors and adapted to varying attack policies through enhanced exploration.
• The system demonstrated robustness in dynamic environments, with the defender maintaining high reward even under evolving attacker strategies.