[Paper Review] The Adversarial Resilience Learning Architecture for AI-based Modelling, Exploration, and Operation of Complex Cyber-Physical Systems

Eric MSP Veith, Nils Wenninghoff+1 more|arXiv (Cornell University)|May 27, 2020

Adversarial Robustness in Machine Learning34 references4 citations

TL;DR

This paper proposes Adversarial Resilience Learning (ARL), a novel AI-based framework that uses competing deep reinforcement learning (DRL) agents—attackers and defenders—to explore, test, and improve the resilience of complex cyber-physical systems (CPS), such as power grids. In a Capture-The-Flag-like tournament, ARL successfully identified attack vectors and generated resilient operational strategies, demonstrating its viability as a benchmark for testing and training robust CPS control systems using diverse DRL algorithms.

ABSTRACT

Modern algorithms in the domain of Deep Reinforcement Learning (DRL) demonstrated remarkable successes; most widely known are those in game-based scenarios, from ATARI video games to Go and the StarCraft~ extsc{II} real-time strategy game. However, applications in the domain of modern Cyber-Physical Systems (CPS) that take advantage a vast variety of DRL algorithms are few. We assume that the benefits would be considerable: Modern CPS have become increasingly complex and evolved beyond traditional methods of modelling and analysis. At the same time, these CPS are confronted with an increasing amount of stochastic inputs, from volatile energy sources in power grids to broad user participation stemming from markets. Approaches of system modelling that use techniques from the domain of Artificial Intelligence (AI) do not focus on analysis and operation. In this paper, we describe the concept of Adversarial Resilience Learning (ARL) that formulates a new approach to complex environment checking and resilient operation: It defines two agent classes, attacker and defender agents. The quintessence of ARL lies in both agents exploring the system and training each other without any domain knowledge. Here, we introduce the ARL software architecture that allows to use a wide range of model-free as well as model-based DRL-based algorithms, and document results of concrete experiment runs on a complex power grid.

Motivation & Objective

• To address the lack of comprehensive, AI-driven methodologies for full-system testing and resilient operation of complex cyber-physical systems (CPS).
• To close the research gap in using AI not only for control but also for systematic exploration, vulnerability detection, and resilience training in CPS.
• To develop a domain-agnostic, software-architected framework that enables the use of diverse model-free and model-based DRL algorithms in adversarial settings.
• To provide a benchmark environment for evaluating and comparing DRL algorithms in realistic, complex CPS scenarios, such as power grids.

Proposed method

• The ARL framework employs two independent DRL agents—attacker and defender—operating in a competitive, co-evolutionary manner within a simulated CPS environment.
• The attacker agent explores the system to discover vulnerabilities and exploit interdependencies, while the defender learns to counteract attacks and maintain system stability.
• The architecture supports integration of various DRL algorithms, including Deep Q-Networks (DQN), Rainbow DQN, Advantage Actor-Critic (A2C), and MuZero, enabling comparative analysis.
• System states are represented through sensor and actuator data, with actions mapped to control decisions on generators and controllable loads.
• The framework uses a Capture-The-Flag (CTF)-like tournament setting where agents compete to control system components, with outcomes averaged over multiple runs.
• The software architecture ensures reproducibility and modularity, allowing plug-in of different DRL algorithms and system models.

Experimental results

Research questions

• RQ1 Can adversarial DRL agents effectively uncover hidden vulnerabilities and interdependencies in complex cyber-physical systems without prior domain knowledge?
• RQ2 How do different DRL algorithms perform in adversarial settings when applied to real-world CPS such as power grids?
• RQ3 To what extent can the defender agent learn resilient operational strategies through co-evolution with an adaptive attacker?
• RQ4 Can the ARL framework serve as a scalable benchmark for evaluating and comparing DRL-based control and analysis methods in CPS?
• RQ5 How do the actions of attacker and defender agents reflect real-world operator behavior and system-relevant node prioritization?

Key findings

• The attacker agent successfully identified and exploited system weaknesses without prior knowledge of the network topology, confirming that adversarial strategies can emerge organically through DRL training.
• The defender agent prioritized control of system-relevant generators and loads, aligning with expert operator behavior and demonstrating effective resilience strategies.
• The ARL framework enabled the use of diverse DRL algorithms—including Rainbow DQN, A2C, and MuZero—within a unified, reproducible architecture for CPS analysis.
• Attackers leveraged simultaneity effects across nodes to maximize disruption, highlighting the importance of temporal coordination in system vulnerabilities.
• The average action distributions across tournaments confirmed that attackers exploit high-impact nodes effectively, while defenders focus on critical infrastructure components.
• The results validate ARL as a viable and effective framework for generating training data, detecting flaws, and improving resilience in complex CPS, particularly in power grids.