[Paper Review] Linear Estimation of Treatment Effects in Demand Response: An Experimental Design Approach

Pan Li, Baosen Zhang|arXiv (Cornell University)|Jun 29, 2017

Smart Grid Energy Management21 references3 citations

TL;DR

This paper proposes a linear estimation framework for average treatment effects (ATE) in demand response programs using experimental design principles. It compares simple linear regression, multiple linear regression, and a modified covariate method, showing that including covariates without modeling their interaction with treatment can reduce estimation accuracy and lead to underestimation of ATE, with the modified covariate method providing more reliable inference when treatment effects are linear in covariates.

ABSTRACT

Demand response aims to stimulate electricity consumers to modify their loads at critical time periods. In this paper, we consider signals in demand response programs as a binary treatment to the customers and estimate the average treatment effect, which is the average change in consumption under the demand response signals. More specifically, we propose to estimate this effect by linear regression models and derive several estimators based on the different models. From both synthetic and real data, we show that including more information about the customers does not always improve estimation accuracy: the interaction between the side information and the demand response signal must be carefully modeled. In addition, we compare the traditional linear regression model with the modified covariate method which models the interaction between treatment effect and covariates. We analyze the variances of these estimators and discuss different cases where each respective estimator works the best. The purpose of these comparisons is not to claim the superiority of the different methods, rather we aim to provide practical guidance on the most suitable estimator to use under different settings. Our results are validated using data collected by Pecan Street and EnergyPlus.

Motivation & Objective

• To estimate the average treatment effect (ATE) of demand response signals on electricity consumption using observational data.
• To evaluate the performance of different linear regression models—simple, multiple, and modified covariate methods—in estimating ATE.
• To provide practical guidance on selecting the most suitable estimator based on data characteristics, treatment assignment probability, and covariate-treatment interaction.
• To demonstrate that including covariates without modeling their interaction with treatment can reduce estimation accuracy and lead to biased ATE estimates.
• To validate the proposed estimators using both synthetic data and real-world datasets from Pecan Street and EnergyPlus.

Proposed method

• Models the demand response signal as a binary treatment and estimates ATE using simple linear regression (SLR), multiple linear regression (MLR), and a modified covariate method (MCM).
• Uses centered treatment indicators (Ti − p) and introduces modified covariates (vi = (Ti − p)xi) to capture interaction effects between treatment and covariates in the MCM.
• Applies t-tests and F-tests to assess statistical significance of treatment effects under each model, with p-values used to evaluate null hypotheses on treatment coefficient significance.
• Derives analytical expressions for estimator variances to compare performance across models under different data conditions.
• Employs both synthetic data and real data from Pecan Street and EnergyPlus to validate estimator behavior and robustness.
• Compares estimation accuracy and significance results across models to identify settings where each estimator performs best.

Experimental results

Research questions

• RQ1 Does including covariates in linear regression models always improve ATE estimation accuracy in demand response studies?
• RQ2 How do different linear regression models—SLR, MLR, and MCM—perform in estimating ATE under varying treatment assignment probabilities and covariate correlations?
• RQ3 Under what conditions does the modified covariate method outperform standard multiple linear regression in ATE estimation?
• RQ4 Can the inclusion of covariates lead to underestimation of ATE, and if so, why?
• RQ5 What role do p-values from t-tests and F-tests play in assessing the reliability of ATE estimates across different modeling approaches?

Key findings

• The multiple linear regression (MLR) estimator produced a significantly lower ATE estimate (0.59) compared to SLR (1.16) and MCM (0.90) on Pecan Street data, suggesting potential underestimation due to poor modeling of treatment-covariate interactions.
• The p-value for the t-test on the treatment coefficient in MLR was 1.4e-2, indicating insignificance at α = 0.01, suggesting MLR may falsely conclude the DR program is ineffective.
• The modified covariate method (MCM) yielded a p-value of 2.9e-09 for the t-test, indicating strong statistical significance of the treatment effect, and its estimate (0.90) was closer to the SLR estimate than MLR.
• The F-test p-values were small across all models, confirming that covariates significantly improve model fit, but this does not imply better inference on treatment effects.
• When treatment effects are linear in covariates, the modified covariate method consistently provides more reliable ATE estimates than standard MLR.
• The study demonstrates that more data or additional covariates do not guarantee better ATE estimation; proper modeling of treatment-covariate interactions is essential.