[Paper Review] A Computational Approach to Identification of Treatment Effects for Policy Evaluation

Sukjin Han, Shenshen Yang|arXiv (Cornell University)|Sep 29, 2020

Healthcare Policy and Management40 references4 citations

TL;DR

This paper proposes a computational framework for deriving sharp nonparametric bounds on policy-relevant treatment effects under binary instruments and unobserved heterogeneity. By leveraging statistical independence of instruments and a flexible set of identifying assumptions—including shape restrictions and exogeneity of covariates—it enables systematic extrapolation of local average treatment effects (LATE) to counterfactual policy settings via linear programming, significantly narrowing identification intervals compared to prior methods.

ABSTRACT

For counterfactual policy evaluation, it is important to ensure that treatment parameters are relevant to policies in question. This is especially challenging under unobserved heterogeneity, as is well featured in the definition of the local average treatment effect (LATE). Being intrinsically local, the LATE is known to lack external validity in counterfactual environments. This paper investigates the possibility of extrapolating local treatment effects to different counterfactual settings when instrumental variables are only binary. We propose a novel framework to systematically calculate sharp nonparametric bounds on various policy-relevant treatment parameters that are defined as weighted averages of the marginal treatment effect (MTE). Our framework is flexible enough to fully incorporate statistical independence (rather than mean independence) of instruments and a large menu of identifying assumptions beyond the shape restrictions on the MTE that have been considered in prior studies. We apply our method to understand the effects of medical insurance policies on the use of medical services.

Motivation & Objective

• To address the lack of external validity in local average treatment effects (LATE) when extrapolating results from randomized experiments to new policy environments.
• To develop a systematic method for calculating sharp nonparametric bounds on treatment parameters defined as weighted averages of the marginal treatment effect (MTE).
• To incorporate statistical independence of instruments (conditional on covariates) and a broad menu of identifying assumptions beyond shape restrictions on the MTE.
• To improve identification precision in settings with discrete instruments by leveraging exogenous covariates and sieve approximations.
• To demonstrate the method’s effectiveness through simulations and applications to medical insurance policy evaluation.

Proposed method

• Formulates policy-relevant treatment parameters as weighted averages of the marginal treatment effect (MTE), including ATE, treatment effect on the treated, and policy-relevant treatment effect (PRTE).
• Uses linear programming to compute sharp nonparametric bounds on these parameters under the assumption of statistical independence between instruments and unobservables, conditional on observed covariates.
• Incorporates a flexible set of identifying assumptions—beyond shape restrictions—such as exogeneity of auxiliary variables and conditional independence, through constraints in the linear program.
• Employs Bernstein polynomial sieve approximations to nonparametrically estimate the MTE function, with convergence properties analyzed under increasing polynomial order K.
• Derives bounds on the MTE function and its weighted averages by solving dual linear programs, ensuring sharpness under the maintained assumptions.
• Validates the method using simulations with varying support of instruments (Z), outcome cardinality (Y), and smoothness of the true MTE function.

Experimental results

Research questions

• RQ1 Can sharp nonparametric bounds on policy-relevant treatment effects be systematically computed when instruments are binary and unobserved heterogeneity is present?
• RQ2 How does statistical independence of instruments (conditional on covariates) improve identification power compared to mean independence in binary instrument settings?
• RQ3 To what extent can auxiliary exogenous variables enhance the precision of bounds on treatment effects in the absence of continuous instruments?
• RQ4 How do shape restrictions and sieve approximations affect the accuracy and robustness of bounds under non-smooth MTE functions?
• RQ5 Can the proposed framework outperform existing partial identification methods, such as Mogstad et al. (2018), in terms of bound sharpness?

Key findings

• The proposed method produces significantly narrower bounds on the average treatment effect (ATE) than Mogstad et al. (2018) in both discrete and continuous outcome settings, especially when the instrument support is expanded.
• Bounds become substantially tighter as the support of the instrument Z increases, confirming that greater exogenous variation improves identification even with binary instruments.
• The inclusion of exogenous covariates—even when not reversely excluded—improves bound sharpness, demonstrating that exogeneity, not exclusion, is the key driver of identification power.
• When the true MTE function is non-smooth (e.g., with kink points like |sin(2πu)|), sieve approximations with low polynomial order K (e.g., K=5) suffer from severe misspecification, but this improves markedly as K increases (e.g., K≥10).
• The Hausdorff distance between the true MTE and the identified set decreases significantly with higher K, indicating reduced misspecification and improved convergence of the approximation.
• The framework successfully identifies sharp bounds on various policy-relevant parameters, including LATE for subgroups and PRTE, under a wide range of identifying assumptions beyond traditional shape restrictions.