[Paper Review] Understanding the Safety Requirements for Learning-based Power Systems Operations

Yize Chen, Daniel Arnold+2 more|arXiv (Cornell University)|Oct 11, 2021

Smart Grid Security and Resilience26 references4 citations

TL;DR

This paper investigates the safety and robustness of reinforcement learning (RL)-based power system operations by demonstrating that minimal adversarial perturbations to system measurements can severely degrade controller performance and violate operational constraints. Through black-box attacks on voltage regulation and topology control tasks, the authors show that RL agents trained on real-world power system data are vulnerable to small input perturbations, leading to unsafe power flows and instability, highlighting the urgent need to integrate security and safety into data-driven grid control design.

ABSTRACT

Recent advancements in machine learning and reinforcement learning have brought increased attention to their applicability in a range of decision-making tasks in the operations of power systems, such as short-term emergency control, Volt/VAr control, long-term residential demand response and battery energy management. Despite the promises of providing strong representation of complex system dynamics and fast, efficient learned operation strategies, the safety requirements of such learning paradigms are less discussed. This paper explores the design requirements on both data and model side of such learning algorithms by exploiting the impacts of adversarial attacks on safety critical system operations. Case studies performed on both voltage regulation and topology control tasks demonstrated the potential vulnerabilities of the standard reinforcement learning algorithms, and possible measures of machine learning robustness and security are discussed for power systems operation tasks.

Motivation & Objective

• To investigate the safety and robustness of learning-based power system operations under adversarial input perturbations.
• To evaluate the vulnerability of reinforcement learning agents in real-world power system tasks such as voltage regulation and topology control.
• To identify critical security gaps in data-driven control systems where model-based safety guarantees are absent.
• To propose a framework for assessing adversarial robustness in power system operations using black-box attack strategies with minimal attacker knowledge.
• To advocate for integrating security and safety objectives into the design of machine learning controllers for power systems.

Proposed method

• Designing a black-box adversarial attack strategy that manipulates system state observations (e.g., load and generation profiles) with minimal perturbations (≤5%) to deceive RL agents.
• Using state estimation and communication network interfaces to inject adversarial noise into real-time measurements without requiring model access.
• Evaluating the impact of perturbations on action outputs and subsequent power flow profiles in both voltage regulation and topology control tasks.
• Employing query-based optimization to generate adversarial examples with low computational overhead (average 0.5s per step).
• Leveraging public testbeds such as the L2RPN and IEEE 39-bus systems to simulate realistic grid dynamics and validate attack effectiveness.
• Releasing a simulation testbed and attack examples on GitHub to support reproducible research in ML security for power systems.

Experimental results

Research questions

• RQ1 How vulnerable are reinforcement learning-based controllers in power system operations to small, adversarial perturbations in input measurements?
• RQ2 What are the physical consequences of adversarial attacks on RL agents in voltage regulation and topology control tasks?
• RQ3 Can a black-box attacker with minimal knowledge (no model access) destabilize a trained RL agent and cause constraint violations?
• RQ4 How do adversarial perturbations affect the decision-making process and action selection in RL agents trained on real power system data?
• RQ5 What design principles and safeguards are needed to enhance the robustness of data-driven power system controllers against such attacks?

Key findings

• Adversarial perturbations of just 5% in load and generation profiles caused reactive power injection decisions from trained RL agents to deviate significantly from optimal values.
• In the IEEE 39-bus voltage regulation task, adversarial attacks led to power flow violations exceeding three times the safe operational range compared to model-based methods.
• The L2RPN test case revealed that similar Q-values for top action sets made RL agents particularly susceptible to state falsification, increasing attack success likelihood.
• Black-box attacks achieved effective manipulation of RL agent behavior with average attack vector computation time under 0.5 seconds per step.
• The study demonstrated that even with minimal attacker knowledge, adversarial inputs can induce cascading instability and constraint violations in real-time power system operations.
• The findings underscore that current RL-based controllers lack inherent safety guarantees and are vulnerable to data-level manipulation, necessitating security-by-design in AI-driven grid operations.