[Paper Review] Multi-Agent Deep Reinforcement Learning for HVAC Control in Commercial Buildings

Liang Yu, Yi Sun+5 more|arXiv (Cornell University)|Jun 25, 2020

Building Energy and Comfort Optimization39 references4 citations

TL;DR

This paper proposes a multi-agent deep reinforcement learning (MADRL) algorithm with an attention mechanism to minimize HVAC energy costs in multi-zone commercial buildings without requiring prior knowledge of thermal dynamics or uncertain parameters. The method achieves 75.25% lower total energy cost compared to baseline methods under real-world occupancy and price variations, demonstrating high effectiveness, robustness, and scalability.

ABSTRACT

In commercial buildings, about 40%-50% of the total electricity consumption is attributed to Heating, Ventilation, and Air Conditioning (HVAC) systems, which places an economic burden on building operators. In this paper, we intend to minimize the energy cost of an HVAC system in a multi-zone commercial building under dynamic pricing with the consideration of random zone occupancy, thermal comfort, and indoor air quality comfort. Due to the existence of unknown thermal dynamics models, parameter uncertainties (e.g., outdoor temperature, electricity price, and number of occupants), spatially and temporally coupled constraints associated with indoor temperature and CO2 concentration, a large discrete solution space, and a non-convex and non-separable objective function, it is very challenging to achieve the above aim. To this end, the above energy cost minimization problem is reformulated as a Markov game. Then, an HVAC control algorithm is proposed to solve the Markov game based on multi-agent deep reinforcement learning with attention mechanism. The proposed algorithm does not require any prior knowledge of uncertain parameters and can operate without knowing building thermal dynamics models. Simulation results based on real-world traces show the effectiveness, robustness and scalability of the proposed algorithm.

Motivation & Objective

• Address the high energy cost of HVAC systems in commercial buildings, which account for 40%-50% of total electricity use.
• Tackle the challenge of minimizing HVAC energy cost under uncertain and dynamic conditions, including random occupancy, fluctuating electricity prices, and indoor air quality constraints.
• Develop a control algorithm that operates without prior knowledge of building thermal dynamics models or uncertain parameters such as outdoor temperature and occupancy levels.
• Ensure thermal and indoor air quality comfort while maintaining system scalability and robustness in complex, coupled environments.

Proposed method

• Reformulate the HVAC control problem as a Markov game to model multi-zone interactions and constraints.
• Design a multi-agent deep reinforcement learning framework with an attention mechanism to enable coordinated decision-making across zones.
• Use the soft actor-critic (SAC) algorithm for training, allowing exploration and stable learning in non-convex, non-separable environments.
• Incorporate state representations that include zone temperature, CO₂ concentration, occupancy, outdoor temperature, and electricity price.
• Implement experience replay buffers with large capacity to support learning in high-dimensional action spaces (e.g., 11^31 actions for 30 zones).
• Train agents end-to-end using real-world building traces, enabling the algorithm to adapt to dynamic and uncertain conditions without model knowledge.

Experimental results

Research questions

• RQ1 Can a multi-agent deep reinforcement learning approach effectively minimize HVAC energy costs in multi-zone commercial buildings without relying on thermal dynamics models?
• RQ2 How does the proposed algorithm perform under random zone occupancy and uncertain parameters such as outdoor temperature and electricity prices?
• RQ3 To what extent does the attention mechanism improve coordination among agents in spatially and temporally coupled HVAC control?
• RQ4 How robust is the algorithm to model uncertainties and disturbances in thermal dynamics?
• RQ5 What is the scalability of the algorithm when the number of zones increases?

Key findings

• The proposed algorithm reduces total energy cost (TEC) by 75.25% compared to a random search (RS) baseline under real-world conditions.
• The algorithm achieves a 56.50% reduction in TEC compared to a heuristic strategy (HS), demonstrating superior performance in cost minimization.
• The algorithm maintains robustness under random disturbances, with average absolute temperature deviation (ATD) ≤1.4°C and average CO₂ deviation (ACD) ≤40 ppm across three disturbance scenarios.
• Convergence is stable over 20,000 training episodes, with the average episode reward increasing and stabilizing over time, indicating effective learning.
• The algorithm scales effectively to 30 zones, with a large action space of 11^31, and maintains performance under high-dimensional control tasks.
• The attention mechanism enables effective coordination among agents, reducing total air supply rate during high-price periods and contributing to energy savings.