[Paper Review] Adaptive Power System Emergency Control using Deep Reinforcement Learning

Qiuhua Huang, Renke Huang+4 more|arXiv (Cornell University)|Mar 9, 2019

Power System Optimization and StabilityEngineering32 references17 citations

TL;DR

This paper proposes deep reinforcement learning (DRL)-based adaptive emergency control for power systems, using end-to-end learning to dynamically respond to disturbances. It introduces the open-source RLGC platform and demonstrates superior robustness and adaptiveness in dynamic braking and under-voltage load shedding across two test systems, outperforming conventional methods even under model uncertainty and noisy observations.

ABSTRACT

Power system emergency control is generally regarded as the last safety net for grid security and resiliency. Existing emergency control schemes are usually designed off-line based on either the conceived "worst" case scenario or a few typical operation scenarios. These schemes are facing significant adaptiveness and robustness issues as increasing uncertainties and variations occur in modern electrical grids. To address these challenges, for the first time, this paper developed novel adaptive emergency control schemes using deep reinforcement learning (DRL), by leveraging the high-dimensional feature extraction and non-linear generalization capabilities of DRL for complex power systems. Furthermore, an open-source platform named RLGC has been designed for the first time to assist the development and benchmarking of DRL algorithms for power system control. Details of the platform and DRL-based emergency control schemes for generator dynamic braking and under-voltage load shedding are presented. Extensive case studies performed in both two-area four-machine system and IEEE 39-Bus system have demonstrated the excellent performance and robustness of the proposed schemes.

Motivation & Objective

• Address the limitations of fixed, off-line emergency control schemes in modern power systems with high uncertainty and variability.
• Develop adaptive, real-time emergency control strategies that dynamically respond to system conditions using deep reinforcement learning.
• Create an open-source benchmarking platform (RLGC) to accelerate research in DRL for power system control.
• Demonstrate robustness of DRL-based schemes to new scenarios, model parameter uncertainty, and observation noise.
• Improve upon conventional methods like Q-learning, MPC, and relay-based control in terms of adaptiveness and performance.

Proposed method

• Formulate power system emergency control as a Markov Decision Process (MDP) with states, actions, and reward functions tailored to transient stability and voltage recovery.
• Employ deep Q-network (DQN) and Proximal Policy Optimization (PPO) algorithms for end-to-end learning from high-dimensional system states.
• Design a custom reward function that prioritizes voltage recovery above standard thresholds while minimizing load shedding.
• Train DRL agents in simulated environments using the two-area, four-machine system and the IEEE 39-bus system.
• Integrate safety constraints via penalty terms in the reward function to prevent violations of operational limits.
• Develop the open-source RLGC platform to support training, testing, and benchmarking of DRL algorithms for power system control.

Experimental results

Research questions

• RQ1 Can DRL-based emergency control adapt effectively to unseen system contingencies and operational conditions?
• RQ2 How does DRL-based control compare to conventional methods (e.g., UVLS relays, MPC) in terms of voltage recovery and load shedding performance?
• RQ3 To what extent is the DRL policy robust to model parameter uncertainty and noisy state observations?
• RQ4 Can DRL agents generalize across different disturbance scenarios without retraining?
• RQ5 How does the RLGC platform enable reproducible and comparable benchmarking of DRL algorithms in power system control?

Key findings

• The DRL-based emergency control scheme achieved faster response times (0.13 s on average) compared to conventional methods, enabling real-time application.
• In the IEEE 39-bus system, the DRL method demonstrated superior robustness to 10% parameter uncertainty and observation noise, maintaining stable voltage recovery.
• Voltage recovery profiles under DRL control met or exceeded the required standard, while shedding significantly less load than MPC and UVLS relay methods.
• The DRL policy generalized well to new contingency scenarios not seen during training, indicating strong generalization capability.
• The RLGC platform successfully enabled consistent training and benchmarking of DRL agents, supporting reproducibility and future research acceleration.
• Compared to conventional Q-learning, the DRL approach showed significantly better performance due to automatic high-dimensional feature extraction and non-linear generalization.