[Paper Review] Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks

Jianhong Wang, Wangkun Xu+3 more|arXiv (Cornell University)|Oct 27, 2021

Optimal Power Flow Distribution75 citations

TL;DR

The paper formulates active voltage control as a Dec-POMDP, builds an open-source MARL benchmark for distribution networks, and evaluates several state-of-the-art MARL algorithms across multiple network scales using real-world data.

ABSTRACT

This paper presents a problem in power networks that creates an exciting and yet challenging real-world scenario for application of multi-agent reinforcement learning (MARL). The emerging trend of decarbonisation is placing excessive stress on power distribution networks. Active voltage control is seen as a promising solution to relieve power congestion and improve voltage quality without extra hardware investment, taking advantage of the controllable apparatuses in the network, such as roof-top photovoltaics (PVs) and static var compensators (SVCs). These controllable apparatuses appear in a vast number and are distributed in a wide geographic area, making MARL a natural candidate. This paper formulates the active voltage control problem in the framework of Dec-POMDP and establishes an open-source environment. It aims to bridge the gap between the power community and the MARL community and be a drive force towards real-world applications of MARL algorithms. Finally, we analyse the special characteristics of the active voltage control problems that cause challenges (e.g. interpretability) for state-of-the-art MARL approaches, and summarise the potential directions.

Motivation & Objective

• Formalize active voltage control as a Dec-POMDP to enable distributed, scalable control.
• Provide an open-source, configurable environment to benchmark MARL for distribution networks.
• Assess how voltage constraints can be handled via barrier-function rewards in MARL.
• Analyze scalability, robustness, and interpretability challenges of MARL in real-world grid settings.

Proposed method

• Formulate the problem as a Dec-POMDP with each PV inverter as an agent operating under partial observations.
• Define a barrier-function reward to encode voltage constraints, including Bowl-shape, L1-shape, and L2-shape variants.
• Construct an open-source Python environment and three scenarios (small to large: 6 to 38 agents) based on real public data.
• Use real-time load and PV data to build realistic, time-varying power injections and voltages.
• Evaluate seven MARL algorithms (IDDPG, MADDPG, COMA, IPPO, MAPPO, SQDDPG, MATD3) under continuous actions.
• Compare MARL performance to traditional methods (OPF and droop control) and analyze barrier-function impacts.

Experimental results

Research questions

• RQ1 Can MARL scale to large distributed networks for active voltage control while maintaining voltage within safety bounds?
• RQ2 How do different voltage barrier functions affect MARL performance in terms of voltage control (CR) and power loss (PL)?
• RQ3 How does MARL compare to centralized OPF and decentralized droop control in this setting?
• RQ4 What challenges (e.g., interpretability, robustness) arise when applying MARL to active voltage control, and how can domain knowledge be integrated?

Key findings

• MADDPG and MATD3 generally perform well across scenarios with different barrier functions.
• Droop control is strong in deployment simplicity but MARL does not consistently outperform it or OPF across all cases.
• The Bowl-shape barrier function preserves high controllable rate while keeping power losses low on smaller networks; barrier function choice critically affects results.
• L1-shape often yields high CR on smaller networks but can incur higher PL on larger networks; L2-shape dominates on the largest network in some setups.
• MARL scales to 33-, 141-, and 322-bus networks with a low control rate requirement, but interpretability and robustness remain key challenges.