[Paper Review] Distributed Multi-Agent Deep Reinforcement Learning Framework for Whole-building HVAC Control

Vinay Hanumaiah, Şahika Genç|arXiv (Cornell University)|Oct 26, 2021

Building Energy and Comfort Optimization4 citations

TL;DR

This paper proposes a scalable, distributed multi-agent deep reinforcement learning (DRL) framework that co-simulates Ray RLlib with EnergyPlus to optimize whole-building HVAC control. By using separate agents for heating and cooling setpoints, the framework achieves over 75% energy reduction while maintaining thermal comfort, with training accelerated across heterogeneous CPUs and GPUs via efficient inter-process synchronization.

ABSTRACT

It is estimated that about 40%-50% of total electricity consumption in commercial buildings can be attributed to Heating, Ventilation, and Air Conditioning (HVAC) systems. Minimizing the energy cost while considering the thermal comfort of the occupants is very challenging due to unknown and complex relationships between various HVAC controls and thermal dynamics inside a building. To this end, we present a multi-agent, distributed deep reinforcement learning (DRL) framework based on Energy Plus simulation environment for optimizing HVAC in commercial buildings. This framework learns the complex thermal dynamics in the building and takes advantage of the differential effect of cooling and heating systems in the building to reduce energy costs, while maintaining the thermal comfort of the occupants. With adaptive penalty, the RL algorithm can be prioritized for energy savings or maintaining thermal comfort. Using DRL, we achieve more than 75\\% savings in energy consumption. The distributed DRL framework can be scaled to multiple GPUs and CPUs of heterogeneous types.

Motivation & Objective

• To address the challenge of high energy consumption in commercial buildings, where HVAC systems account for up to 50% of total energy use.
• To overcome the limitations of rule-based and model-based control methods by enabling model-free, adaptive optimization through reinforcement learning.
• To develop a scalable, distributed DRL framework that supports multi-zone and multi-agent control for complex thermal dynamics in large buildings.
• To demonstrate the effectiveness of the framework under diverse weather conditions and varying trade-offs between energy savings and thermal comfort.
• To optimize training cost and convergence time using heterogeneous cloud instances (e.g., AWS) for practical deployment.

Proposed method

• The framework uses a co-simulation interface between Ray RLlib and EnergyPlus, where EnergyPlus simulates building thermal dynamics and provides state observations to an OpenAI Gym environment.
• Reinforcement learning agents in RLlib act based on observations, receive rewards based on energy use and thermal comfort, and update policies via policy gradient algorithms (e.g., PPO, APEX-DDPG).
• A differential penalty coefficient (α) balances energy cost and thermal comfort, enabling trade-off analysis between energy savings and temperature violations.
• Multi-agent control is implemented with separate neural networks for heating and cooling setpoints, allowing specialized optimization for each mode.
• Distributed training is enabled via RLlib’s multi-worker architecture, with inter-process communication managed through EnergyPlus callbacks and message queues.
• The system is deployed on AWS cloud instances, with training cost and performance evaluated across GPU and CPU configurations.

Experimental results

Research questions

• RQ1 How does the distributed DRL framework perform in reducing HVAC energy consumption across different weather conditions?
• RQ2 What is the impact of the energy-temperature penalty coefficient (α) on the trade-off between energy savings and thermal comfort?
• RQ3 How does multi-agent control (separate agents for heating and cooling) compare to multi-zone control in terms of reward, energy use, and temperature violations?
• RQ4 Which DRL algorithm (PPO vs. APEX-DDPG) achieves faster convergence and lower training time with comparable performance?
• RQ5 What is the optimal combination of cloud hardware (CPU/GPU) for minimizing training cost and time in HVAC DRL training?

Key findings

• The framework achieves over 75% reduction in HVAC energy consumption compared to baseline rule-based control.
• Multi-agent control outperforms multi-zone control across all α values, showing improved reward, lower energy use, and reduced temperature violations.
• APEX-DDPG converges 84% faster than PPO and reduces training time by 60%, with comparable reward values.
• The ml.g4dn.16xlarge AWS instance provides the lowest training cost ($1.36) and fastest convergence (15 minutes) among tested configurations.
• The system scales efficiently across heterogeneous CPUs and GPUs, enabling distributed training with minimal latency via callback-based synchronization.
• Weather conditions significantly affect performance: cities with higher humidity show lower temperature violations due to more stable outdoor temperatures.