[Paper Review] ModelicaGym: Applying Reinforcement Learning to Modelica Models

Oleh Lukianykhin, Tetiana Bogodorova|arXiv (Cornell University)|Sep 18, 2019

Modeling and Simulation Systems11 references4 citations

TL;DR

ModelicaGym is an open-source toolbox that integrates Modelica models via Functional Mock-up Units (FMUs) into the OpenAI Gym reinforcement learning (RL) framework, enabling seamless RL application to dynamic system models. It successfully enables Q-learning training on a Cart-Pole benchmark, demonstrating effective control with optimized hyperparameters like time step and reward shaping.

ABSTRACT

This paper presents ModelicaGym toolbox that was developed to employ Reinforcement Learning (RL) for solving optimization and control tasks in Modelica models. The developed tool allows connecting models using Functional Mock-up Interface (FMI) toOpenAI Gym toolkit in order to exploit Modelica equation-based modelling and co-simulation together with RL algorithms as a functionality of the tools correspondingly. Thus, ModelicaGym facilitates fast and convenient development of RL algorithms and their comparison when solving optimal control problem for Modelicadynamic models. Inheritance structure ofModelicaGymtoolbox's classes and the implemented methods are discussed in details. The toolbox functionality validation is performed on Cart-Pole balancing problem. This includes physical system model description and its integration using the toolbox, experiments on selection and influence of the model parameters (i.e. force magnitude, Cart-pole mass ratio, reward ratio, and simulation time step) on the learning process of Q-learning algorithm supported with the discussion of the simulation results.

Motivation & Objective

• To bridge the gap between Modelica-based system modeling and reinforcement learning (RL) by enabling direct integration of Modelica models into the OpenAI Gym RL environment.
• To address the lack of a generalized, reusable tool for connecting Modelica FMUs with RL frameworks, avoiding boilerplate code and enabling plug-and-play RL experimentation.
• To support both open-source and proprietary Modelica tools (e.g., OpenModelica, Dymola) through FMI-compliant FMUs, enhancing tool-agnostic model deployment.
• To validate the toolbox’s functionality and usability on a standard RL benchmark—Cart-Pole balancing—ensuring reproducibility and performance evaluation.
• To provide a modular, extensible, and scalable framework for future RL applications in complex engineering systems, such as power systems or building energy models.

Proposed method

• The toolbox uses the Functional Mock-up Interface (FMI) standard to export Modelica models as Functional Mock-up Units (FMUs), enabling interoperability with external simulation and control environments.
• It implements a Gym environment wrapper that interfaces with PyFMI to load and execute FMUs, translating Modelica model dynamics into a standard RL environment API (reset, step, render).
• The system supports configurable environment parameters such as force magnitude, mass ratios, reward shaping (positive/negative rewards), and simulation time step.
• The toolbox is implemented in Python, leveraging the OpenAI Gym API for consistent RL agent interaction and enabling plug-and-play testing of RL algorithms.
• It supports both co-simulation and model-exchange FMU modes, with extensibility for future support of model-exchange mode upon community demand.
• The architecture is object-oriented, with clear class inheritance, enabling modular extension and reuse across different Modelica-based control problems.

Experimental results

Research questions

• RQ1 Can a generic, reusable toolbox be developed to integrate Modelica models into the OpenAI Gym RL framework via FMI?
• RQ2 How do key model and training parameters—such as force magnitude, mass ratio, reward shaping, and time step—affect the learning performance of a Q-learning agent on a Cart-Pole system?
• RQ3 What is the optimal balance between simulation time step and training efficiency while maintaining stable and effective control in the Cart-Pole environment?
• RQ4 To what extent does the magnitude of negative rewards influence the convergence speed and stability of the RL training process?
• RQ5 Can the toolbox support both open-source and proprietary Modelica tools through standardized FMUs, ensuring broad usability across engineering workflows?

Key findings

• The ModelicaGym toolbox successfully enables end-to-end reinforcement learning on a Modelica-based Cart-Pole model, with Q-learning achieving stable control when hyperparameters are properly tuned.
• A simulation time step of 0.1 seconds was found to provide the best trade-off between training stability and execution efficiency, outperforming both smaller (0.01 s) and larger (0.5 s or 1 s) steps.
• Experiments showed that a larger negative reward magnitude significantly improved learning speed and performance, while smaller magnitudes led to slow convergence or performance degradation.
• Training with a time step of 0.5 s or 1 s failed to achieve the required simulation length (four steps), indicating insufficient control frequency to stabilize the pole.
• Execution time per simulation step remained nearly constant across different parameter sets, indicating that the performance overhead is primarily due to FMU call frequency rather than model complexity.
• The toolbox demonstrated modularity and extensibility, supporting integration with both open-source (e.g., OpenModelica) and proprietary (e.g., Dymola) Modelica tools via FMI-compliant FMUs.