[Paper Review] Exact P-values for Network Interference

Susan Athey, Dean Eckles+1 more|arXiv (Cornell University)|Jun 5, 2015

Advanced Causal Inference Techniques21 references4 citations

TL;DR

This paper develops exact p-values for non-sharp null hypotheses in network interference experiments by constructing an artificial experiment where the null becomes sharp, enabling valid randomization inference. It provides exact p-values for hypotheses restricting spillovers by distance, edge category, or peer effect heterogeneity, correcting flawed asymptotic approximations in prior work.

ABSTRACT

We study the calculation of exact p-values for a large class of non-sharp null hypotheses about treatment effects in a setting with data from experiments involving members of a single connected network. The class includes null hypotheses that limit the effect of one unit's treatment status on another according to the distance between units; for example, the hypothesis might specify that the treatment status of immediate neighbors has no effect, or that units more than two edges away have no effect. We also consider hypotheses concerning the validity of sparsification of a network (for example based on the strength of ties) and hypotheses restricting heterogeneity in peer effects (so that, for example, only the number or fraction treated among neighboring units matters). Our general approach is to define an artificial experiment, such that the null hypothesis that was not sharp for the original experiment is sharp for the artificial experiment, and such that the randomization analysis for the artificial experiment is validated by the design of the original experiment.

Motivation & Objective

• To address the lack of exact p-values for non-sharp null hypotheses in network interference experiments where spillovers depend on network distance or edge structure.
• To correct flawed asymptotic inference methods that underestimate Type I error rates, such as the incorrect variance assumption in Bond et al. (2012).
• To enable valid randomization inference under hypotheses restricting spillovers to neighbors within a certain distance, specific edge categories, or homogeneous peer effects.
• To generalize randomization inference to complex interference structures without relying on large sample approximations or strong parametric assumptions.

Proposed method

• Construct an artificial experiment where the original non-sharp null becomes sharp, preserving the design's validity for randomization inference.
• Define the randomization distribution over potential outcomes under the artificial experiment, ensuring it aligns with the original experiment’s design.
• Use the potential outcomes framework to model interference via a network structure, with edges categorized (e.g., strong/weak) or weighted.
• Derive exact p-values by computing the proportion of randomizations under the artificial experiment that yield test statistics as extreme as the observed one.
• Apply the method to three classes of null hypotheses: distance-based spillovers, edge-category restrictions, and peer effect heterogeneity.
• Derive a score test statistic based on the covariance between residuals under the null and the expected number of treated neighbors, ensuring exactness under the artificial design.

Experimental results

Research questions

• RQ1 Can exact p-values be computed for non-sharp null hypotheses about treatment spillovers in a single connected network?
• RQ2 What is the correct randomization distribution for testing hypotheses that only immediate neighbors (or neighbors within k steps) affect outcomes?
• RQ3 How can one test whether network sparsification—e.g., removing weak ties—invalidates the assumption of no spillovers beyond a threshold?
• RQ4 What is the correct test statistic and sampling distribution for hypotheses restricting peer effects to the number or fraction of treated neighbors?
• RQ5 Why do standard asymptotic methods, such as those in Bond et al. (2012), lead to inflated Type I error rates in network interference settings?

Key findings

• The incorrect variance assumption in Bond et al. (2012), which treats the test statistic as having variance 0.5, leads to a Type I error rate of 0.157 under the null, not the nominal 0.05.
• A simulation study confirms this inflated error rate, with a rejection rate of 0.153 (95% CI: 0.144–0.163), closely matching the theoretical value.
• The proposed method yields exact p-values by constructing an artificial experiment where the null becomes sharp, ensuring valid randomization inference.
• The score test statistic is proportional to the covariance between residuals under the null and the number of treated neighbors, providing a robust and exact test for peer effects.
• The method applies generally to hypotheses restricting interference by distance, edge category, or heterogeneity in peer effects, without requiring large sample approximations.
• The approach corrects for the bias in prior asymptotic methods, particularly in settings with strong network dependence and non-ignorable spillovers.