[Paper Review] Multi-agent Reinforcement Learning Embedded Game for the Optimization of Building Energy Control and Power System Planning

Jun Hao|arXiv (Cornell University)|Jan 17, 2019

Smart Grid Energy Management61 references6 citations

TL;DR

This paper proposes a multi-agent reinforcement learning (MARL)-embedded game-theoretic framework to optimize HVAC control and power system planning in academic and commercial buildings. By modeling building managers as players in a game that minimizes social cost—combining energy expenses and productivity loss—the approach uses real-time pricing and RL to achieve Nash equilibrium faster than traditional game theory, reducing energy costs and maintaining indoor comfort.

ABSTRACT

Most of the current game-theoretic demand-side management methods focus primarily on the scheduling of home appliances, and the related numerical experiments are analyzed under various scenarios to achieve the corresponding Nash-equilibrium (NE) and optimal results. However, not much work is conducted for academic or commercial buildings. The methods for optimizing academic-buildings are distinct from the optimal methods for home appliances. In my study, we address a novel methodology to control the operation of heating, ventilation, and air conditioning system (HVAC). With the development of Artificial Intelligence and computer technologies, reinforcement learning (RL) can be implemented in multiple realistic scenarios and help people to solve thousands of real-world problems. Reinforcement Learning, which is considered as the art of future AI, builds the bridge between agents and environments through Markov Decision Chain or Neural Network and has seldom been used in power system. The art of RL is that once the simulator for a specific environment is built, the algorithm can keep learning from the environment. Therefore, RL is capable of dealing with constantly changing simulator inputs such as power demand, the condition of power system and outdoor temperature, etc. Compared with the existing distribution power system planning mechanisms and the related game theoretical methodologies, our proposed algorithm can plan and optimize the hourly energy usage, and have the ability to corporate with even shorter time window if needed.

Motivation & Objective

• To address the gap in demand-side management for academic and commercial buildings, where HVAC systems dominate energy use and impact occupant productivity.
• To develop a control strategy that minimizes social cost—defined as the sum of energy expenditure and monetary value of reduced working productivity—rather than focusing solely on energy or cost reduction.
• To model building managers as rational agents in a non-cooperative game, where their strategies are indoor temperature settings to optimize individual and collective outcomes.
• To overcome the computational inefficiency of traditional game-theoretic methods in large-scale systems by integrating reinforcement learning (RL) for faster convergence to Nash equilibrium.
• To demonstrate the framework on the University of Denver campus smart grid, using real-time pricing and dynamic occupancy data for realistic simulation.

Proposed method

• Formulates a non-cooperative game where each building manager is a player, and their strategy is setting the indoor temperature to minimize personal social cost.
• Defines social cost as the sum of energy cost (based on real-time locational marginal pricing) and productivity loss (monetized based on thermal comfort impact).
• Uses a Markov Decision Process (MDP) framework with state representation including outdoor temperature, occupancy levels, and real-time electricity prices.
• Employs Q-learning-based multi-agent reinforcement learning (MARL) to enable agents to learn optimal temperature settings through interaction with the environment and feedback on social cost.
• Integrates a parallel computing scheme to simulate the smart grid environment and accelerate training and decision-making.
• Applies a novel optimization problem (Ψ) to prove that the Nash equilibrium of the game corresponds to the global minimum of social cost, ensuring optimality.

Experimental results

Research questions

• RQ1 Can a game-theoretic approach effectively balance energy cost and occupant productivity in academic and commercial buildings with high HVAC energy use?
• RQ2 How does the integration of reinforcement learning improve the computational efficiency of achieving Nash equilibrium in large-scale building energy control games?
• RQ3 To what extent can real-time pricing and occupancy data be leveraged to optimize HVAC scheduling while maintaining thermal comfort?
• RQ4 Does the proposed MARL-embedded game framework achieve lower social cost and faster convergence compared to traditional game-theoretic methods?
• RQ5 Is the Nash equilibrium of the proposed game equivalent to the global minimum of the social cost function?

Key findings

• The proposed MARL-embedded game-theoretic framework achieves optimal social cost minimization at Nash equilibrium, with theoretical proof that the equilibrium corresponds to the global minimum of the social cost function.
• Reinforcement learning reduces computational time significantly compared to traditional game-theoretic methods, especially as the number of buildings or strategies increases.
• The simulation results show that the algorithm successfully adjusts indoor temperatures in response to real-time electricity prices and occupancy levels, maintaining thermal comfort within acceptable ranges.
• Energy costs per building and overall campus electricity charges are reduced compared to baseline strategies, without compromising occupant productivity.
• The framework demonstrates scalability and adaptability, capable of handling shorter time windows and dynamic changes in power system conditions, such as fluctuating real-time pricing and renewable generation.