[Paper Review] Controlling Commercial Cooling Systems Using Reinforcement Learning

Jerry Luo, Cosmin Păduraru+34 more|arXiv (Cornell University)|Nov 11, 2022

Smart Grid Energy Management22 citations

TL;DR

The paper presents BCOOLER, an RL-based controller for commercial chiller plants that learns from offline and live data to reduce energy consumption while respecting safety and operational constraints, achieving 9% and 13% energy savings in two live deployments.

ABSTRACT

This paper is a technical overview of DeepMind and Google's recent work on reinforcement learning for controlling commercial cooling systems. Building on expertise that began with cooling Google's data centers more efficiently, we recently conducted live experiments on two real-world facilities in partnership with Trane Technologies, a building management system provider. These live experiments had a variety of challenges in areas such as evaluation, learning from offline data, and constraint satisfaction. Our paper describes these challenges in the hope that awareness of them will benefit future applied RL work. We also describe the way we adapted our RL system to deal with these challenges, resulting in energy savings of approximately 9% and 13% respectively at the two live experiment sites.

Motivation & Objective

• Demonstrate reinforcement learning for real-time supervisory control of commercial chiller plants.
• Address challenges of learning from limited data, constraint satisfaction, and non-stationary dynamics in live facilities.
• Show energy efficiency gains compared to heuristic SOO controllers in two real buildings.
• Describe practical adaptations enabling RL to operate safely within industry constraints.

Proposed method

• Formulate chiller plant control as a constrained MDP with 50 state features and 12 action setpoints.
• Develop BCOOLER, an ensemble-based, constrained RL learner that predicts energy use and observation constraint violations.
• Use Monte Carlo-style targets for energy and constraint predictions with a multi-headed neural network.
• Employ an ensemble of networks to estimate uncertainty and drive exploration with constraint filtering.
• Perform 5-minute timestep decisions with action pruning and mode-aware action masking to enable real-time inference.
• Retrain the model daily on offline and online data to adapt to changing conditions.

Experimental results

Research questions

• RQ1 Can an RL-based controller improve energy efficiency of commercial chiller plants while respecting safety and operational constraints?
• RQ2 What challenges arise when deploying RL in real buildings, and how can they be mitigated (data scarcity, constraints, non-stationarity)?
• RQ3 How does BCOOLER compare to tradition SOO-based heuristics in real-world deployments?
• RQ4 What practices enable scalable deployment of RL across different buildings and operating modes?

Key findings

• BCOOLER achieved 9% energy savings in one live site and 13% in another compared with the SOO baseline.
• The RL system maintained occupant comfort by adhering to observed constraints similar to the baseline.
• Performance improved over time, likely due to more data and software refinements.
• RF-based ensemble approach provided uncertainty estimates used to balance exploration and constraint satisfaction.
• The method handles multiple chillers and a large action space with action pruning and mode-specific masking.
• Results suggest potential for greater gains when paired with facility pre-commissioning and AI-readiness efforts.