[Paper Review] Analyzing multilevel experiments in the presence of peer effects

Guillaume Basse, Avi Feller|arXiv (Cornell University)|Aug 24, 2016

Advanced Causal Inference Techniques3 citations

TL;DR

This paper proposes unbiased estimators for multilevel experiments with varying household sizes, comparing individual- and household-weighted estimands, and demonstrates equivalence between linear regression and randomization inference when using appropriate standard errors. It finds substantial spillover effects in a real-world absenteeism intervention, validating the method through simulations and empirical analysis.

ABSTRACT

Multilevel or two-stage randomization is a powerful design for estimating treatment effects in the presence of social interactions. Our motivating example is a multilevel randomized trial evaluating an intervention to reduce student absenteeism in the School District of Philadelphia. In that experiment, households with multiple students were first assigned to treatment or control; then, in treated households, one student was randomly assigned to treatment. Using this example, we highlight key considerations for analyzing multilevel experiments in practice. Our first contribution is to address additional complexities that arise when household sizes vary; in this case, researchers must decide between assigning equal weight to households or equal weight to individuals. We propose unbiased estimators for a broad class of individual- and household-weighted estimands, with corresponding theoretical and estimated variances. Our second contribution is to connect two common approaches for analyzing multilevel designs: linear regression and randomization inference. We show that, with suitably chosen standard errors, these two approaches yield identical point and variance estimates, which is somewhat surprising given the complex randomization scheme. Finally, we explore options for incorporating covariates to improve precision and confirm our analytic results via simulation studies. We apply these methods to the attendance study and find large, substantively meaningful spillover effects.

Motivation & Objective

• To address analytical challenges in multilevel randomized trials when household sizes vary, particularly the choice between equal weighting of households or individuals.
• To develop unbiased estimators for both individual- and household-weighted estimands, with corresponding theoretical and estimated variances.
• To establish a formal connection between linear regression and randomization inference in complex multilevel designs with peer effects.
• To evaluate covariate adjustment strategies for improving precision in multilevel experimental designs.
• To validate the proposed methods through simulation studies and application to a real-world attendance intervention in Philadelphia.

Proposed method

• Proposes unbiased estimators for individual- and household-weighted estimands in multilevel experiments with heterogeneous household sizes.
• Derives theoretical and estimated variances for these estimators to support inference.
• Demonstrates that linear regression with carefully chosen robust standard errors produces identical point and variance estimates as randomization inference under the complex randomization scheme.
• Uses simulation studies to confirm the validity and precision gains of the proposed estimators under various design conditions.
• Incorporates covariate adjustment to improve estimation precision, particularly in settings with heterogeneous treatment effects.
• Applies the methods to a real multilevel randomized trial on student absenteeism in Philadelphia, using household-level and individual-level randomization.

Experimental results

Research questions

• RQ1 How should researchers handle unequal household sizes in multilevel experiments when estimating treatment effects with peer effects?
• RQ2 What is the relationship between linear regression and randomization inference in multilevel designs with complex randomization schemes?
• RQ3 Can covariate adjustment improve precision in multilevel experiments with peer effects, and how should it be implemented?
• RQ4 What are the implications of choosing individual-weighted versus household-weighted estimands in terms of bias and variance?
• RQ5 What magnitude and direction of spillover effects can be detected in real-world multilevel experiments with social interactions?

Key findings

• The proposed unbiased estimators for both individual- and household-weighted estimands yield valid inference with correct theoretical and estimated variances.
• Linear regression with appropriate robust standard errors produces identical point and variance estimates as randomization inference, despite the complex randomization mechanism.
• Covariate adjustment improves precision in the estimation of treatment effects in multilevel experiments.
• Simulation studies confirm the validity and robustness of the proposed estimators under various design and outcome configurations.
• Empirical application to the Philadelphia absenteeism trial reveals large, substantively meaningful spillover effects, indicating strong peer effects in educational outcomes.