[Paper Review] Energy-Efficient Thermal Comfort Control in Smart Buildings via Deep Reinforcement Learning

Guanyu Gao, Jie Li+1 more|arXiv (Cornell University)|Jan 15, 2019

Building Energy and Comfort Optimization38 references70 citations

TL;DR

The paper presents a deep reinforcement learning framework using Deep Deterministic Policy Gradients (DDPG) for continuous HVAC control, coupled with a Bayesian-regularized neural network to predict thermal comfort, evaluated in a TRNSYS-based building simulator to reduce energy use while maintaining occupant comfort.

ABSTRACT

Heating, Ventilation, and Air Conditioning (HVAC) is extremely energy-consuming, accounting for 40% of total building energy consumption. Therefore, it is crucial to design some energy-efficient building thermal control policies which can reduce the energy consumption of HVAC while maintaining the comfort of the occupants. However, implementing such a policy is challenging, because it involves various influencing factors in a building environment, which are usually hard to model and may be different from case to case. To address this challenge, we propose a deep reinforcement learning based framework for energy optimization and thermal comfort control in smart buildings. We formulate the building thermal control as a cost-minimization problem which jointly considers the energy consumption of HVAC and the thermal comfort of the occupants. To solve the problem, we first adopt a deep neural network based approach for predicting the occupants' thermal comfort, and then adopt Deep Deterministic Policy Gradients (DDPG) for learning the thermal control policy. To evaluate the performance, we implement a building thermal control simulation system and evaluate the performance under various settings. The experiment results show that our method can improve the thermal comfort prediction accuracy, and reduce the energy consumption of HVAC while improving the occupants' thermal comfort.

Motivation & Objective

• Reduce HVAC energy consumption while maintaining occupant thermal comfort in smart buildings.
• Develop a predictive model for occupant thermal comfort incorporating multiple influencing factors.
• Leverage deep reinforcement learning with continuous actions for precise HVAC set-point control.
• Validate the approach in a building simulation environment under varied conditions.

Proposed method

• Develop a Bayesian-regularized feedforward neural network to predict occupants’ thermal comfort from indoor state variables.
• Formulate energy optimization and thermal comfort as a Markov Decision Process with a cost (reward) function combining energy use and comfort penalties.
• Apply Deep Deterministic Policy Gradients (DDPG) with an actor-critic architecture for continuous set-point control of temperature and humidity.
• Train the DDPG agent in a TRNSYS-based building simulation, using a replay buffer and Ornstein-Uhlenbeck exploration noise.
• Use a reward function that penalizes HVAC energy consumption and discomfort outside an acceptable thermal comfort threshold (M within [-D, D]).

Experimental results

Research questions

• RQ1 Can continuous-action DDPG control policies reduce HVAC energy while maintaining occupant comfort compared to baselines?
• RQ2 How accurately can thermal comfort be predicted from indoor environmental variables using Bayesian-regularized neural networks?
• RQ3 What is the impact of the energy-comfort weighting parameter on the learned policy and overall performance?
• RQ4 Does integrating a learned thermal comfort predictor as feedback improve control decisions over model-based approaches?

Key findings

• The proposed method integrates a DNN-based thermal comfort predictor with DDPG for joint energy optimization and comfort control.
• The neural network predictor uses Bayesian regularization to improve generalization in comfort estimation.
• The system is evaluated in a TRNSYS-based simulation showing the approach can reduce HVAC energy consumption while maintaining or improving occupant comfort.
• The action space remains continuous for precise HVAC set-points, avoiding discretization limits of alternative DRL methods.
• A configurable trade-off parameter balances energy cost against comfort penalties in the reward, enabling customization to occupant needs.