[Paper Review] Learning to run a power network challenge for training topology controllers

Antoine Marot, Benjamin Donnot+5 more|arXiv (Cornell University)|Dec 5, 2019

Optimal Power Flow Distribution15 references60 citations

TL;DR

The paper introduces a framework to learn topology-control agents for power grids via imitation and reinforcement learning, and reports on the first Learning to Run a Power Network (L2RPN) competition using an IEEE14-based environment, plus an oracle upper-bound analysis revealing optimization gaps.

ABSTRACT

For power grid operations, a large body of research focuses on using generation redispatching, load shedding or demand side management flexibilities. However, a less costly and potentially more flexible option would be grid topology reconfiguration, as already partially exploited by Coreso (European RSC) and RTE (French TSO) operations. Beyond previous work on branch switching, bus reconfigurations are a broader class of action and could provide some substantial benefits to route electricity and optimize the grid capacity to keep it within safety margins. Because of its non-linear and combinatorial nature, no existing optimal power flow solver can yet tackle this problem. We here propose a new framework to learn topology controllers through imitation and reinforcement learning. We present the design and the results of the first "Learning to Run a Power Network" challenge released with this framework. We finally develop a method providing performance upper-bounds (oracle), which highlights remaining unsolved challenges and suggests future directions of improvement.

Motivation & Objective

• Motivate topology reconfiguration as a flexible, cost-effective means to keep grids within safety margins amid renewable integration and demand variability.
• Provide a platform to learn and benchmark topology-control agents operating in real-time power system environments.
• Explore the feasibility and robustness of reinforcement learning and related AI methods for continuous, near-real-time grid optimization.
• Offer an oracle-based upper bound to quantify how close submitted agents come to optimal topology-control performance.

Proposed method

• Open-source synthetic power grid environment based on IEEE14 with topology actions (line switching and bus switching) and real-time injections.
• Formulation of the problem as a two-factor Markov Decision Process to capture unobserved inputs, observed injections, topology, and outputs (flows and overloads).
• Definition of a real-time reward/score function aimed at maximizing residual grid capacity while penalizing illegal actions and cascading failures.
• Implementation and evaluation of diverse agents (reinforcement learning, expert systems, and tree search) within a 6-week Codalab competition.
• Analysis of agent behaviors, action-space reduction strategies, and robustness; comparison to an oracle path-based upper bound for performance bounds.
• Use of A3C (actor-critic) and dueling DQN architectures among top entrants, with curriculum learning and expert-rule guidance to improve stability and performance.

Experimental results

Research questions

• RQ1 Can RL and related AI methods learn effective topology-control policies for real-time grid operation under safety and operational constraints?
• RQ2 What is the impact of action-space design and expert priors on learning efficiency and policy robustness in topology reconfiguration tasks?
• RQ3 How close can learned agents approach optimal topology trajectories, and what is the gap to an oracle upper bound?
• RQ4 What characteristics of scenarios (difficulty, duration, overload timing) challenge current approaches most?
• RQ5 Does a challenge-based benchmark provide a productive platform for advancing topology-control research in power systems?

Key findings

• RL-based approaches achieved the best results among participants in the competition.
• Top teams included LB (dueling DQN + prior analysis) and LR (A3C-based actor-critic) with open-source codes; LR relied on learning without querying simulations at test time.
• An oracle trajectory-based upper bound revealed a remaining optimization gap between the best agents and the optimal solution.
• Most participants limited exploration, with some agents focusing on a small subset of substations, highlighting exploration vs. exploitation trade-offs.
• The study demonstrates feasibility of topology-control through AI and identifies robustness and scalability gaps for future benchmarks.
• The analysis shows that while top agents can finish test scenarios, several challenging scenarios remain where no agent succeeds, motivating larger-scale benchmarks.