[Paper Review] Differentially Private Estimation of Heterogeneous Causal Effects

Fengshi Niu, Harsha Nori+4 more|arXiv (Cornell University)|Feb 22, 2022

Advanced Causal Inference Techniques5 citations

TL;DR

This paper introduces a general meta-algorithm for differentially private estimation of heterogeneous causal effects using sample splitting and parallel composition to tightly bound privacy loss. It demonstrates that most accuracy loss under differential privacy stems from increased variance rather than bias, especially in flexible multi-stage estimators like DR- and R-learners, with performance degrading significantly under strong privacy (ε=1) and small sample sizes.

ABSTRACT

Estimating heterogeneous treatment effects in domains such as healthcare or social science often involves sensitive data where protecting privacy is important. We introduce a general meta-algorithm for estimating conditional average treatment effects (CATE) with differential privacy (DP) guarantees. Our meta-algorithm can work with simple, single-stage CATE estimators such as S-learner and more complex multi-stage estimators such as DR and R-learner. We perform a tight privacy analysis by taking advantage of sample splitting in our meta-algorithm and the parallel composition property of differential privacy. In this paper, we implement our approach using DP-EBMs as the base learner. DP-EBMs are interpretable, high-accuracy models with privacy guarantees, which allow us to directly observe the impact of DP noise on the learned causal model. Our experiments show that multi-stage CATE estimators incur larger accuracy loss than single-stage CATE or ATE estimators and that most of the accuracy loss from differential privacy is due to an increase in variance, not biased estimates of treatment effects.

Motivation & Objective

• To develop a general, privacy-preserving meta-algorithm for estimating conditional average treatment effects (CATE) with differential privacy guarantees.
• To enable the use of both single-stage (e.g., S-learner) and multi-stage (e.g., DR, R-learner) CATE estimators under differential privacy.
• To analyze the trade-off between privacy (ε) and estimation accuracy, particularly focusing on variance and bias components.
• To evaluate the impact of differential privacy on flexible, interpretable models like DP-EBMs in causal inference settings.
• To empirically quantify how sample size and privacy budget affect the performance of different CATE estimation methods.

Proposed method

• The meta-algorithm uses sample splitting to decouple estimation of nuisance parameters and treatment effects, enabling parallel composition of differential privacy.
• It applies differential privacy to base learners (e.g., DP-EBMs) used in CATE estimation, ensuring end-to-end privacy guarantees.
• The method supports various CATE estimators, including S-learner (single-stage), DR-learner, and R-learner (multi-stage), via modular integration.
• Privacy budget is conserved by leveraging parallel composition across split data subsets, reducing cumulative privacy loss.
• The approach uses DP-EBMs—interpretable, high-accuracy models with built-in differential privacy—for transparent noise impact observation.
• Estimation accuracy is evaluated via mean squared error (MSE), decomposed into bias and variance components to isolate privacy effects.

Experimental results

Research questions

• RQ1 How can a general meta-algorithm be designed to provide differential privacy guarantees for both single-stage and multi-stage CATE estimators?
• RQ2 To what extent does differential privacy increase estimation variance versus bias in CATE estimation?
• RQ3 How does the performance of flexible multi-stage CATE estimators (e.g., DR, R-learner) degrade under strong privacy constraints (e.g., ε=1) compared to simpler estimators?
• RQ4 What sample size is required for flexible CATE estimators to outperform simple ATE estimators under varying privacy budgets?
• RQ5 How does the privacy-accuracy trade-off differ between interpretable, private models like DP-EBMs and standard machine learning models in causal inference?

Key findings

• At ε=1 and n=500, the MSE of DP-EBM-S-learner remained stable at approximately 0.016, close to the true ATE MSE, indicating robustness for simple ATE estimation under strong privacy.
• For flexible estimators like DP-EBM-DR-learner and DP-EBM-R-learner, MSE increased by a factor of 1000 from ε=16 to ε=1 when n=500, showing high sensitivity to privacy constraints at small sample sizes.
• At n=32000, the same privacy transition (ε=16 to ε=1) increased MSE by less than a factor of 10, indicating diminishing returns in accuracy loss at larger sample sizes.
• The bias of the DP-EBM-DR-learner changed by at most a factor of 2 when moving from ε=16 to ε=1, while variance increased by a factor of about 10, indicating variance dominates accuracy loss.
• Flexible CATE estimators outperform ATE estimators only when sample size exceeds ~20,000 under strong privacy (ε=1), highlighting a significant data cost for privacy.
• The study confirms that the privacy-accuracy trade-off in CATE estimation is primarily driven by increased variance due to DP noise, not biased estimation, supporting the use of private CATE models in high-stakes domains.