[Paper Review] Difference-in-Differences for Policy Evaluation

Brantly Callaway|arXiv (Cornell University)|Jan 1, 2022

Advanced Causal Inference Techniques4 citations

TL;DR

This paper reviews recent advances in difference-in-differences (DID) methodology, highlighting limitations of two-way fixed effects (TWFE) regressions under treatment effect heterogeneity and varying treatment timing. It proposes alternative estimators that are more robust and easier to implement, offering improved causal inference in policy evaluation with empirical illustrations and sensitivity analyses to assess parallel trends violations.

ABSTRACT

Difference-in-differences is one of the most used identification strategies in empirical work in economics. This chapter reviews a number of important, recent developments related to difference-in-differences. First, this chapter reviews recent work pointing out limitations of two way fixed effects regressions (these are panel data regressions that have been the dominant approach to implementing difference-in-differences identification strategies) that arise in empirically relevant settings where there are more than two time periods, variation in treatment timing across units, and treatment effect heterogeneity. Second, this chapter reviews recently proposed alternative approaches that are able to circumvent these issues without being substantially more complicated to implement. Third, this chapter covers a number of extensions to these results, paying particular attention to (i) parallel trends assumptions that hold only after conditioning on observed covariates and (ii) strategies to partially identify causal effect parameters in difference-in-differences applications in cases where the parallel trends assumption may be violated.

Motivation & Objective

• To identify and address critical limitations of two-way fixed effects (TWFE) regressions in difference-in-differences (DID) applications under treatment effect heterogeneity.
• To present and evaluate recently developed alternative estimators that are more robust to heterogeneity and varying treatment timing.
• To examine the role of parallel trends assumptions, especially when they hold only after conditioning on observed covariates.
• To explore strategies for partial identification of causal effects when the parallel trends assumption may be violated.
• To demonstrate practical implications through empirical applications and sensitivity analyses, particularly in minimum wage and teen employment studies.

Proposed method

• Proposes alternative DID estimators that decompose treatment effects by time since treatment, avoiding the weighting issues inherent in TWFE regressions.
• Uses event study estimators with proper weighting schemes to recover dynamic treatment effects, even under heterogeneity.
• Applies sensitivity analysis techniques to assess robustness of results to violations of the parallel trends assumption.
• Employs counterfactual path estimation under the parallel trends assumption, conditional on observed covariates when necessary.
• Utilizes software-implemented methods such as the callaway and sant'anna (2021) estimator for staggered adoption designs.
• Combines pre-treatment trend analysis with formal sensitivity analysis to evaluate the plausibility of parallel trends.

Experimental results

Research questions

• RQ1 How do two-way fixed effects regressions perform under treatment effect heterogeneity and staggered treatment timing?
• RQ2 What are the key limitations of TWFE regressions in DID applications with multiple time periods and heterogeneous treatment effects?
• RQ3 Can alternative estimators provide more reliable causal estimates than TWFE regressions in settings with treatment heterogeneity?
• RQ4 How can researchers assess the robustness of DID results when the parallel trends assumption may be violated?
• RQ5 To what extent do estimation weights in TWFE regressions lead to misleading comparisons, especially with early and late adopters?

Key findings

• TWFE regressions are not robust to treatment effect heterogeneity when there are more than two time periods and staggered treatment timing, leading to biased estimators that combine effects with potentially misleading weights.
• Alternative estimators such as the event study and group-specific DID methods provide more accurate and interpretable estimates of treatment effects under heterogeneity.
• In the minimum wage application, results were sensitive to whether 2007 data were included, indicating that treatment effect heterogeneity significantly affects estimated impacts.
• Sensitivity analysis showed that estimated effects on teen employment were not robust to violations of parallel trends as large as those observed in pre-treatment periods, with the confidence set ranging from -0.073 to 0.023.
• A non-negligible amount of weight was placed on 'bad comparisons'—using already-treated units as controls—highlighting the need for careful design choices such as excluding early-treated units or post-treatment periods.
• Despite similar point estimates in some cases, differences between TWFE and newer methods were large enough to suggest that researchers should prefer the newer, more robust approaches.