[Paper Review] The Bayesian Synthetic Control: Improved Counterfactual Estimation in the Social Sciences through Probabilistic Modeling

Elias Tuomaala|arXiv (Cornell University)|Oct 14, 2019

Advanced Causal Inference Techniques13 references4 citations

TL;DR

The paper proposes the Bayesian Synthetic Control (BSC), a probabilistic framework that improves counterfactual estimation in social science by replacing the frequentist synthetic control method (SCM) with a hierarchical Bayesian model. BSC uses Markov Chain Monte Carlo (MCMC) sampling to estimate latent factors and explicitly models uncertainty through posterior distributions, enabling valid finite-sample credible intervals and reducing overfitting, with empirical applications showing improved predictive accuracy and differing conclusions on statistical significance compared to SCM.

ABSTRACT

Social scientists often study how a policy reform impacted a single targeted country. Increasingly, this is done with the synthetic control method (SCM). SCM models the country's counterfactual (non-reform or untreated) trajectory as a weighted average of other countries' outcomes. The method struggles to quantify uncertainty; eg. it cannot produce confidence intervals. It is also suspect to overfit. We propose an alternative method, the Bayesian synthetic control (BSC), which lacks these flaws. Using MCMC sampling, we implement the method for two previously studied datasets. The proposed method outperforms SCM in a simple test of predictive accuracy and casts some doubt on significance of prior findings. The studied reforms are the German reunification of 1990 and the California tobacco legislation of 1988. BSC borrows its causal model, the linear latent factor model, from the SCM literature. Unlike SCM, BSC estimates the latent factors explicitly through a dimensionality reduction. All uncertainty is captured in the posterior distribution so that, unlike for SCM, credible intervals are easily derived. Further, BSC's reliability on the target panel dataset can be assessed through a posterior predictive check; SCM and its frequentist derivatives use up the required information while testing statistical significance.

Motivation & Objective

• To address the limitations of the synthetic control method (SCM), particularly its inability to quantify uncertainty and susceptibility to overfitting.
• To develop a Bayesian framework that models treatment effects with full posterior inference, enabling credible intervals and uncertainty quantification in finite samples.
• To improve model validation through posterior predictive checks and relabeling techniques, which are not available in frequentist SCM.
• To demonstrate the method’s practical utility through two real-world applications: German reunification (1990) and California’s Proposition 99 (1988).
• To show that BSC can yield different statistical significance conclusions than prior SCM-based studies, highlighting methodological sensitivity in causal inference.

Proposed method

• BSC adopts a linear latent factor model as the causal structure, shared with SCM, but estimates latent factors explicitly via dimensionality reduction.
• The method uses a hierarchical Bayesian model to jointly estimate treatment effects and latent factor trajectories, incorporating prior distributions to regularize estimation.
• Markov Chain Monte Carlo (MCMC) sampling is employed to draw from the full posterior distribution, enabling uncertainty quantification and credible interval computation.
• Model validity is assessed via posterior predictive checks and relabeling techniques, which test whether the model can reproduce observed data under the fitted posterior.
• The number of latent factors is selected using WAIC (Watanabe-Akaike Information Criterion), though the paper notes Bayesian model averaging would be preferable.
• The framework naturally accommodates missing data by treating missing values as additional treated units, enhancing data flexibility.

Experimental results

Research questions

• RQ1 Does the Bayesian Synthetic Control method produce more accurate counterfactual predictions than the frequentist synthetic control method in real-world applications?
• RQ2 Can the Bayesian approach provide valid finite-sample credible intervals for treatment effects, unlike SCM?
• RQ3 How does the BSC method’s assessment of statistical significance compare to prior SCM-based findings for the German reunification and California Proposition 99 cases?
• RQ4 To what extent can posterior predictive checks and relabeling improve model validation in synthetic control settings?
• RQ5 Can the BSC framework be extended to handle multiple treated units or missing data without reworking the implementation?

Key findings

• In the German reunification case, BSC outperformed SCM in a simple predictive accuracy test, suggesting potential overfitting in the frequentist approach.
• For California’s Proposition 99, BSC found the treatment effect to be insignificant for most of the post-reform period, contradicting Abadie et al. (2015) but aligning with Ben-Michael et al. (2018) in rejecting significance.
• By year 2000, BSC found the treatment effect to be just barely significant, indicating a cumulative effect that is statistically detectable only over time.
• The posterior predictive checks in BSC revealed that the model could reproduce observed data reasonably well, supporting model validity in the German reunification case.
• BSC’s credible intervals were valid in finite samples, unlike the asymptotic confidence intervals assumed in SCM, making it more reliable for small-sample policy evaluation.
• The method demonstrated computational feasibility and flexibility, including the ability to handle missing data by treating them as separate treated units.