[Paper Review] Data-Driven Demand-Side Flexibility Quantification: Prediction and Approximation of Flexibility Envelopes

Nami Hekmat, Hanmin Cai+3 more|arXiv (Cornell University)|Oct 25, 2021

Smart Grid Energy Management7 references4 citations

TL;DR

This paper proposes a data-driven pipeline to reduce computational and communication overhead in real-time demand-side flexibility quantification by replacing computationally intensive optimization with machine learning prediction and probabilistic approximation of flexibility envelopes. Using AdaBoost for prediction and a 2D skewed normal distribution for approximation, the method reduces computational load by a factor of 9 and communication load by 6.6x, with high accuracy (R² > 0.94).

ABSTRACT

Real-time quantification of residential building energy flexibility is needed to enable a cost-efficient operation of active distribution grids. A promising means is to use the so-called flexibility envelope concept to represent the time-dependent and inter-temporally coupled flexibility potential. However, existing optimization-based quantification entails high computational burdens limiting flexibility utilization in real-time applications, and a more computationally efficient quantification approach is desired. Additionally, the communication of a flexibility envelope to system operators in its original form is data-intensive. In order to address the computational burdens, this paper first trains several machine learning models based on historical quantification results for online use. Subsequently, probability distribution functions are proposed to approximate the flexibility envelopes with significantly fewer parameters, which can be communicated to system operators instead of the original flexibility envelope. The results show that the most promising prediction and approximation approaches allow for a minimum reduction of the computational burden by a factor of 9 and of the communication load by a factor of 6.6, respectively.

Motivation & Objective

• Address the high computational burden of real-time flexibility envelope quantification using optimization-based methods.
• Reduce the data-intensive communication of flexibility envelopes to distribution system operators.
• Enable scalable, real-time coordination of residential building flexibility in active distribution grids.
• Develop a practical, low-latency pipeline for online flexibility quantification and reporting.
• Improve scalability by replacing repeated optimization with trained machine learning models and compact probabilistic representations.

Proposed method

• Train machine learning models (AdaBoost, SVM, KNN, SVR, GMM) on historical flexibility envelope data to predict future envelopes online.
• Use probability distribution functions—specifically 2D and 3D variants of normal, skewed normal, and Gaussian mixture distributions—to approximate the flexibility envelope with minimal parameters.
• Apply a two-stage pipeline: first predict the envelope using ML models, then approximate it using the best-fitting PDF to minimize data transmission.
• Optimize model selection using metrics like mean absolute error (MAE) and R² score to evaluate prediction and approximation accuracy.
• Integrate the best-performing prediction and approximation models into a hybrid pipeline for end-to-end efficiency.
• Use a benchmark model (direct optimization) as a baseline for performance comparison.

Experimental results

Research questions

• RQ1 Can machine learning models accurately predict future flexibility envelopes with significantly reduced computational cost compared to repeated optimization?
• RQ2 Can probability distribution functions effectively approximate the full flexibility envelope using only a few hyperparameters?
• RQ3 What is the trade-off between prediction accuracy and communication efficiency when replacing raw envelope data with parametric PDFs?
• RQ4 How do different ML models and PDF variants compare in terms of prediction and approximation performance?
• RQ5 Can a hybrid pipeline combining the best prediction and approximation models achieve both low computational load and high fidelity?

Key findings

• The AdaBoost-based prediction model reduces computational burden by a factor of 9 compared to the original optimization-based method.
• The 2D skewed normal distribution (SND) approximation reduces communication load by a factor of 6.6 compared to transmitting the full envelope.
• The best prediction model (AdaBoost) achieves an MAE of 21 minutes and an R² score of 0.95, significantly outperforming the benchmark (MAE: 93 min, R²: 0.79).
• The best approximation model (2D-SND) achieves an R² score of 0.99 and an MAE of 25 minutes, demonstrating high fidelity with minimal parameters.
• The combined pipeline (AdaBoost prediction + 2D-SND approximation) achieves an MAE of 35 minutes and an R² score of 0.94, showing that errors do not accumulate additively.
• A hybrid model that switches between AdaBoost, SVM, and KNN per power level improves prediction accuracy slightly, achieving an MAE of 31 minutes and an R² of 0.95.