[Paper Review] Multi-Agent Reinforcement Learning for Power Grid Topology Optimization

Erica van der Sar, Alessandro Zocca+1 more|arXiv (Cornell University)|Oct 4, 2023

Optimal Power Flow Distribution4 citations

TL;DR

This paper proposes a hierarchical multi-agent reinforcement learning (MARL) framework for power grid topology optimization using bus-switching actions to enhance grid reliability and scalability. By decomposing control across multiple agents per substation and leveraging SACD and PPO algorithms, the approach achieves optimal performance comparable to single-agent methods while improving stability and adaptability to network changes.

ABSTRACT

Recent challenges in operating power networks arise from increasing energy demands and unpredictable renewable sources like wind and solar. While reinforcement learning (RL) shows promise in managing these networks, through topological actions like bus and line switching, efficiently handling large action spaces as networks grow is crucial. This paper presents a hierarchical multi-agent reinforcement learning (MARL) framework tailored for these expansive action spaces, leveraging the power grid's inherent hierarchical nature. Experimental results indicate the MARL framework's competitive performance with single-agent RL methods. We also compare different RL algorithms for lower-level agents alongside different policies for higher-order agents.

Motivation & Objective

• Address the scalability challenge of reinforcement learning (RL) in large power grids with vast combinatorial action spaces due to bus and line switching.
• Overcome the limitations of single-agent RL and rule-based or after-state RL methods in adapting to dynamic grid conditions.
• Develop a cooperative multi-agent framework that leverages the inherent hierarchical structure of transmission networks for improved control and scalability.
• Compare different MARL architectures and RL algorithms (SACD, PPO) at the low-level agent, and evaluate mid-level policy strategies (CAPA, fixed, random) for coordination.
• Demonstrate that multi-agent RL can achieve optimal grid performance with enhanced stability and adaptability compared to single-agent baselines.

Proposed method

• Implement a three-tiered hierarchical MARL architecture: high-level rule-based agents for global coordination, mid-level agents selecting which substation to act on, and low-level agents performing bus-switching actions.
• Use graph neural networks (GNNs) to encode substation topology and power flow states, enabling agents to reason over graph-structured grid data.
• Deploy SACD (Soft Actor-Critic Discrete) and PPO (Proximal Policy Optimization) as low-level RL algorithms, with centralized training and decentralized execution (CTDE) to improve sample efficiency.
• Introduce two multi-agent variants: Independent PPO (IPPO) and Decentralized PPO (DPPO), and two SACD variants: Independent SACD (ISACD) and Decentralized SACD (DSACD), to evaluate coordination strategies.
• Apply hyperparameter optimization via Optuna to tune parameters for multi-agent agents, improving performance over direct transfer from single-agent configurations.
• Use the Grid2Op environment to simulate realistic power grid operations, including topological changes and contingency handling, with rewards based on grid security and load flow stability.

Experimental results

Research questions

• RQ1 Can a hierarchical multi-agent RL framework achieve performance comparable to single-agent RL in power grid topology optimization while improving scalability?
• RQ2 How do different low-level RL algorithms (SACD vs. PPO) perform in a multi-agent setting when applied to grid topology control?
• RQ3 What impact do different mid-level coordination policies (CAPA, fixed, random) have on the learning stability and final performance of the MARL system?
• RQ4 To what extent does hyperparameter tuning improve the performance of multi-agent RL agents compared to transferring parameters from single-agent setups?
• RQ5 Does the decentralized multi-agent approach enhance adaptability to network changes compared to single-agent or after-state RL methods?

Key findings

• The decentralized SACD (DSACD) agent with hyperparameter tuning achieved the optimal score after a number of environment interactions comparable to the single-agent SACD, while maintaining significantly greater training stability.
• The ISACD agent with tuned hyperparameters reached peak performance but exhibited instability, underscoring the sensitivity of SACD to hyperparameter choices in multi-agent settings.
• The PPO-based multi-agent agents (IPPO and DPPO) showed less sensitivity to hyperparameter differences than SACD, with both achieving optimal performance after a similar number of interactions as the single-agent PPO.
• The mid-level CAPA policy outperformed both the fixed and random policies in terms of learning stability and final performance, with the random policy leading to the most unstable training dynamics.
• The multi-agent framework successfully reduced the effective action space per agent, enhancing scalability and enabling easier integration of complex actions like generator redispatching.
• The hierarchical MARL framework demonstrated competitive performance with single-agent RL while offering improved robustness and adaptability, particularly in dynamic grid conditions.