[Paper Review] Real-world challenges for multi-agent reinforcement learning in grid-interactive buildings

Kingsley Nweye, Bo Liu+2 more|arXiv (Cornell University)|Nov 25, 2021

Smart Parking Systems Research4 citations

TL;DR

This paper identifies nine real-world challenges for multi-agent reinforcement learning (MARL) in grid-interactive buildings, proposing a standardized framework to improve benchmarking and reproducibility. It demonstrates that longer offline training from an optimized rule-based controller (RBC) significantly improves long-term energy flexibility performance, while information sharing among agents provides no measurable benefit in the tested setup.

ABSTRACT

Building upon prior research that highlighted the need for standardizing environments for building control research, and inspired by recently introduced challenges for real life reinforcement learning control, here we propose a non-exhaustive set of nine real world challenges for reinforcement learning control in grid-interactive buildings. We argue that research in this area should be expressed in this framework in addition to providing a standardized environment for repeatability. Advanced controllers such as model predictive control and reinforcement learning (RL) control have both advantages and disadvantages that prevent them from being implemented in real world problems. Comparisons between the two are rare, and often biased. By focusing on the challenges, we can investigate the performance of the controllers under a variety of situations and generate a fair comparison. As a demonstration, we implement the offline learning challenge in CityLearn and study the impact of different levels of domain knowledge and complexity of RL algorithms. We show that the sequence of operations utilized in a rule based controller (RBC) used for offline training affects the performance of the RL agents when evaluated on a set of four energy flexibility metrics. Longer offline learning from an optimized RBC leads to improved performance in the long run. RL agents that learn from a simplified RBC risk poorer performance as the offline learning period increases. We also observe no impact on performance from information sharing amongst agents. We call for a more interdisciplinary effort of the research community to address the real world challenges, and unlock the potential of grid-interactive building

Motivation & Objective

• To address the lack of standardized benchmarks for evaluating reinforcement learning (RL) in grid-interactive buildings.
• To identify and formalize real-world challenges that hinder the practical deployment of MARL in building control.
• To enable fair, systematic comparisons between RL and traditional control methods like model predictive control (MPC) and rule-based controllers (RBC).
• To demonstrate the impact of offline training quality and RBC design on subsequent RL agent performance.
• To advocate for interdisciplinary collaboration between building science and machine learning communities to accelerate real-world deployment.

Proposed method

• Proposes a non-exhaustive framework of nine real-world challenges (C1–C9) for MARL in grid-interactive buildings, focusing on practical deployment barriers.
• Uses the CityLearn environment to implement and evaluate the offline learning challenge (C8), simulating real-world training scenarios.
• Trains RL agents (SAC and MARLISA) on data generated by two RBCs: a basic version and an optimized version with improved operational logic.
• Evaluates agents on four energy flexibility metrics: district average daily peak, load factor, ramping, and net electricity consumption.
• Employs a common interface (OpenAI Gym) and open-source implementation to ensure reproducibility and community-wide benchmarking.
• Analyzes the impact of offline training duration, RBC sequence of operations, and agent information sharing on final performance.

Experimental results

Research questions

• RQ1 How does the quality of offline pre-training data—specifically, the sequence of operations in a rule-based controller—affect the performance of subsequently trained RL agents?
• RQ2 Does longer offline training from a high-quality RBC lead to better long-term energy flexibility performance in grid-interactive buildings?
• RQ3 What is the impact of information sharing among multi-agent RL algorithms on performance across key energy flexibility metrics?
• RQ4 Can simpler RL algorithms (e.g., SAC) outperform more complex MARL architectures in real-world building control scenarios?
• RQ5 To what extent do existing RBCs already provide significant energy flexibility, reducing the marginal benefit of deploying advanced RL controllers?

Key findings

• Longer offline training from an optimized RBC leads to superior long-term performance in reducing district average daily peak, load factor, and ramping, despite slower initial convergence.
• RL agents trained on a simplified RBC show progressively worse performance as the offline training period increases, indicating that poor-quality pre-training data degrades final outcomes.
• No significant performance difference was observed between the SAC and MARLISA RL algorithms across the four energy flexibility metrics, suggesting that the added complexity of information sharing provides no benefit in this setting.
• Net electricity consumption remained largely unchanged across all configurations, indicating that the RBC design—particularly its focus on night-time charging using high COP heat pumps—already captures much of the available flexibility.
• The optimized RBC significantly outperformed all RL controllers in reducing peak demand and improving load factor, highlighting that improving existing control logic may be more effective than deploying complex RL systems.
• The results suggest that the quality of pre-training data is more critical than algorithmic complexity, and that high-fidelity RBCs can serve as strong baselines for real-world deployment.