[Paper Review] Towards a Scalable and Flexible Simulation and Testing Environment Toolbox for Intelligent Microgrid Control

Henrik Bode, Stefan Heid+3 more|arXiv (Cornell University)|May 11, 2020

Microgrid Control and Optimization25 references4 citations

TL;DR

This paper introduces the OpenModelica Microgrid Gym (OMG) toolbox, an open-source, scalable simulation environment for intelligent microgrid control that integrates OpenModelica for modeling and a Python-based OpenAI Gym interface for plug-and-play testing of control algorithms, including reinforcement learning. The key contribution is a flexible, extensible platform enabling standardized evaluation and safe optimization of control strategies across diverse microgrid topologies and operating conditions.

ABSTRACT

Micro- and smart grids (MSG) play an important role both for integrating renewable energy sources in conventional electricity grids and for providing power supply in remote areas. Modern MSGs are largely driven by power electronic converters due to their high efficiency and flexibility. Nevertheless, controlling MSGs is a challenging task due to highest requirements on energy availability, safety and voltage quality within a wide range of different MSG topologies. This results in a high demand for comprehensive testing of new control concepts during their development phase and comparisons with the state of the art in order to ensure their feasibility. This applies in particular to data-driven control approaches from the field of reinforcement learning (RL), whose stability and operating behavior can hardly be evaluated a priori. Therefore, the OpenModelica Microgrid Gym (OMG) package, an open-source software toolbox for the simulation and control optimization of MSGs, is proposed. It is capable of modeling and simulating arbitrary MSG topologies and offers a Python-based interface for plug \& play controller testing. In particular, the standardized OpenAI Gym interface allows for easy RL-based controller integration. Besides the presentation of the OMG toolbox, application examples are highlighted including safe Bayesian optimization for low-level controller tuning.

Motivation & Objective

• Address the lack of standardized, open, and extensible platforms for testing and comparing intelligent microgrid control algorithms across diverse topologies and operating conditions.
• Enable safe and efficient data-driven optimization of low-level controller parameters (e.g., PID gains) in microgrids using Bayesian optimization with safety constraints.
• Provide a plug-and-play interface between high-fidelity microgrid models (via OpenModelica) and control algorithms implemented in Python, particularly for reinforcement learning.
• Support comprehensive validation of control strategies under varying system configurations, including hybrid AC/DC microgrids and stochastic load/feed-in conditions.
• Facilitate reproducible benchmarking and development of novel control approaches by offering a standardized, open-source simulation and testing framework.

Proposed method

• Leverages OpenModelica as a backend for modeling arbitrary microgrid topologies, including power electronic converters, inverters, storage, and loads.
• Exposes a Python-based interface using the OpenAI Gym standard, enabling seamless integration of control algorithms, especially reinforcement learning agents.
• Implements a safe Bayesian optimization framework using Gaussian processes to tune PID controller parameters (Kp, Ki) while enforcing safety constraints on performance metrics.
• Employs a performance metric (J) as a function of controller gains, with safety thresholds defined to prevent system instability or component overloading.
• Uses phase-locked loops (PLLs) and auxiliary control components within the OpenModelica model to ensure accurate synchronization and stability during simulation.
• Applies a multi-dimensional parameter search space for controller tuning, with visualization of performance landscapes and confidence bounds to guide optimization.

Experimental results

Research questions

• RQ1 Can a standardized, open-source simulation and testing platform be developed to enable fair and reproducible comparison of intelligent microgrid control strategies across diverse topologies?
• RQ2 How effectively can data-driven optimization methods, such as Safe Bayesian Optimization, ensure safety during the tuning of low-level microgrid controllers without prior knowledge of system dynamics?
• RQ3 To what extent does the performance of a PID controller vary across different gain combinations, and can this be reliably modeled and optimized using Gaussian processes?
• RQ4 What are the limitations of standard Gaussian process-based uncertainty modeling when applied to microgrid control performance functions with abrupt changes or plateaus?
• RQ5 Can the integration of expert knowledge into data-driven control optimization improve safety and convergence in reinforcement learning-based microgrid control?

Key findings

• The OMG toolbox successfully enables simulation and testing of arbitrary microgrid topologies using OpenModelica, with full interoperability via a standardized Python interface.
• Safe Bayesian optimization using Gaussian processes could not reliably predict performance platetus or rapid drops in the performance landscape due to mismatched covariance assumptions, leading to overconfidence or missed constraints.
• The optimal PID controller gains found were Kp = 0.0125 V/A and Ki = 117.81 V/(A·s), achieving fast current response with minimal overshoot and no violation of nominal current limits.
• The performance function of the controller exhibited regions of high constancy and sharp transitions, which the standard GP covariance structure failed to capture accurately, limiting optimization reliability.
• Despite the use of safety thresholds (Jmin = -1.06), the GP-based optimization still risked unsafe parameter regions due to poor modeling of abrupt performance changes.
• The study highlights that safe, data-driven microgrid control remains a significant open challenge, especially when relying solely on generic uncertainty models like Gaussian processes without expert-informed priors.