[Paper Review] Towards Optimal District Heating Temperature Control in China with Deep Reinforcement Learning

Adrien Le-Coz, Tahar Nabil+1 more|arXiv (Cornell University)|Dec 17, 2020

Building Energy and Comfort Optimization17 references4 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL) approach to optimize district heating supply water temperature in China’s secondary networks, using a recurrent neural network (RNN) to predict indoor temperatures and train DRL agents. The agents achieve 2.19% energy savings and 495 tons of CO₂ reduction per season compared to an optimized baseline, while improving thermal comfort.

ABSTRACT

Achieving efficiency gains in Chinese district heating networks, thereby reducing their carbon footprint, requires new optimal control methods going beyond current industry tools. Focusing on the secondary network, we propose a data-driven deep reinforcement learning (DRL) approach to address this task. We build a recurrent neural network, trained on simulated data, to predict the indoor temperatures. This model is then used to train two DRL agents, with or without expert guidance, for the optimal control of the supply water temperature. Our tests in a multi-apartment setting show that both agents can ensure a higher thermal comfort and at the same time a smaller energy cost, compared to an optimized baseline strategy.

Motivation & Objective

• To address energy inefficiencies in Chinese district heating networks, which are major contributors to carbon emissions due to outdated control strategies.
• To develop a data-driven, optimal control method for secondary network temperature regulation that improves thermal comfort and reduces energy consumption.
• To evaluate whether deep reinforcement learning can outperform traditional water curve-based control strategies in real-world-like simulations.
• To assess the impact of expert-guided action space on DRL agent performance and deployment feasibility.
• To demonstrate that a reward function based solely on thermal comfort deviation can yield energy savings without explicit energy cost weighting.

Proposed method

• A recurrent neural network (RNN) is trained on simulated data to predict indoor temperatures based on supply temperature, outdoor temperature, time of day, and historical measurements.
• Two DRL agents are trained using Deep Q-Networks (DQN): Agent 1 uses incremental actions relative to the previous supply temperature, while Agent 2 uses deviations from a learned baseline strategy.
• The environment is modeled as a Markov Decision Process with a state vector including outdoor temperature, supply temperature, time of day, and indoor temperatures over a 24-hour history.
• The reward function penalizes absolute deviations from a target indoor temperature (18 °C), with no explicit energy cost weighting: r = -Σ|T_in - T_target|.
• Training uses weather data from seven Chinese cities to avoid overfitting, with testing conducted on an eighth city (Yuncheng) to evaluate generalization.
• The baseline strategy uses an optimized water curve, which is manually tuned and represents current industrial practice.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning outperform the current industry-standard water curve strategy in terms of energy efficiency and thermal comfort in Chinese district heating systems?
• RQ2 How does expert-guided action space (Agent 2) affect the performance and stability of DRL agents in real-world deployment scenarios?
• RQ3 To what extent can a reward function based solely on thermal comfort deviation lead to reduced energy consumption without explicit energy cost penalties?
• RQ4 How does the RNN-based temperature prediction model contribute to the robustness and generalization of the DRL agents across different climatic conditions?
• RQ5 What are the potential CO₂ and energy savings when applying DRL to a large-scale district heating network serving one million square meters?

Key findings

• Agent 2 achieved 2.19% energy savings and 495 tons of CO₂ reduction per season compared to the optimized baseline strategy in a multi-apartment simulation.
• Both DRL agents reduced mean absolute error in indoor temperature by 8.5% compared to the baseline, with Agent 2 achieving the lowest error (0.545 °C) and standard deviation (0.692 °C).
• The reward function based solely on thermal comfort deviation proved more stable and robust than weighted trade-off functions, while still yielding significant energy savings.
• Expert-guided action space (Agent 2) improved performance and ensured actions remained within 3 °C of the expert baseline, increasing operator trust and deployment feasibility.
• The PID controller performed slightly better than the baseline but was outperformed by both DRL agents in both energy efficiency and thermal comfort.
• The results suggest that DRL can effectively learn complex thermal dynamics in district heating systems, even in low-flexibility environments with fixed heat prices and no storage.