[Paper Review] Policy design in experiments with unknown interference

Davide Viviano, Rudder, Jess|arXiv (Cornell University)|Nov 16, 2020

Advanced Causal Inference Techniques4 citations

TL;DR

This paper proposes experimental designs for policy evaluation under unknown interference, where units are grouped in clusters with unobserved network interactions. It introduces a single-wave experiment to estimate marginal treatment effects accounting for spillovers and a multiple-wave experiment to learn welfare-maximizing treatment rules, with theoretical guarantees and empirical validation in a large-scale field experiment.

ABSTRACT

This paper studies experimental designs for estimation and inference on policies with spillover effects. Units are organized into a finite number of large clusters and interact in unknown ways within each cluster. First, we introduce a single-wave experiment that, by varying the randomization across cluster pairs, estimates the marginal effect of a change in treatment probabilities, taking spillover effects into account. Using the marginal effect, we propose a test for policy optimality. Second, we design a multiple-wave experiment to estimate welfare-maximizing treatment rules. We provide strong theoretical guarantees and an implementation in a large-scale field experiment.

Motivation & Objective

• To address the challenge of designing policies when interference effects are present but network structures are unknown or unmeasurable.
• To develop experimental methods that estimate welfare-maximizing treatment rules despite limited knowledge of spillover patterns.
• To provide theoretical guarantees for inference and policy learning in settings with unobserved network interference.
• To validate the proposed methods through implementation in a large-scale field experiment with real-world data.
• To bridge the gap between standard treatment effect estimation and welfare-maximizing policy design under interference.

Proposed method

• Introduces a single-wave experiment that varies treatment probabilities across cluster pairs to estimate marginal effects while accounting for spillovers.
• Uses the estimated marginal effect to construct a formal test for policy optimality, testing whether increasing treatment probability improves welfare.
• Proposes a multiple-wave experimental design that adaptively learns the optimal treatment rule through sequential learning and regret minimization.
• Employs a rescaling scheme in the multi-wave design with convergence guarantees, using adaptive learning rates of order $1/t$ and $1/\sqrt{t}$.
• Calibrates spillover effects using empirical data from prior studies (e.g., Cai et al., 2015) to simulate realistic interference patterns.
• Applies the methods in a large-scale field experiment in Pakistan, with IRB approval and registration, using data from Precision Development and the Center for Economic Research in Pakistan.

Experimental results

Research questions

• RQ1 How can experimental designs estimate welfare-maximizing treatment rules when interference networks are unknown and unmeasured?
• RQ2 What experimental design enables valid inference on the marginal effect of treatment probability changes while accounting for spillovers?
• RQ3 Can a multi-wave experimental design learn the optimal treatment rule with theoretical guarantees on regret and convergence?
• RQ4 How does the power of the policy optimality test vary with cluster size and number of clusters?
• RQ5 What is the relative welfare improvement from using the proposed adaptive learning method compared to baseline strategies?

Key findings

• The single-wave experiment achieves 90–97% coverage for 5% size tests across different cluster sizes and numbers of clusters, with power increasing in cluster size and number of clusters.
• In the multi-wave experiment, the method achieves an average out-of-sample welfare improvement of up to 36.0% relative to the best competitor when $n=600$, $K=20$, and $T=20$.
• The worst-case in-sample regret across clusters is reduced to 0.294–0.387, indicating robust performance even in the most adverse cluster configurations.
• The adaptive learning rule with $1/t$ rescaling achieves faster convergence and lower regret than non-adaptive $1/\sqrt{t}$ rules in simulation experiments.
• The relative welfare improvement from increasing treatment probability by 10% is highest under strong spillovers ($\alpha=0.4$), with up to 24.4% improvement upon rejection of the null hypothesis.
• The method successfully identifies optimal treatment rules in simulations calibrated to real-world data from Alatas et al. (2012) and Cai et al. (2015), demonstrating practical relevance.