[Paper Review] Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs

Ruoxuan Xiong, Alex Chin+1 more|arXiv (Cornell University)|Jun 10, 2024

Advanced Statistical Process MonitoringDecision Sciences3 citations

TL;DR

This paper proposes an empirical Bayes design for switchback experiments on single aggregate units, accounting for carryover effects, periodicity, serial correlation, and simultaneous interventions. By leveraging prior experimental data to optimize interval lengths and switching patterns, the method reduces mean squared error (MSE) by 33% compared to the status quo in a real ride-sharing platform case study.

ABSTRACT

We study the design and analysis of switchback experiments conducted on a single aggregate unit. The design problem is to partition the continuous time space into intervals and switch treatments between intervals, in order to minimize the estimation error of the treatment effect. We show that the estimation error depends on four factors: carryover effects, periodicity, serially correlated outcomes, and impacts from simultaneous experiments. We derive a rigorous bias-variance decomposition and show the tradeoffs of the estimation error from these factors. The decomposition provides three new insights in choosing a design: First, balancing the periodicity between treated and control intervals reduces the variance; second, switching less frequently reduces the bias from carryover effects while increasing the variance from correlated outcomes, and vice versa; third, randomizing interval start and end points reduces both bias and variance from simultaneous experiments. Combining these insights, we propose a new empirical Bayes design approach. This approach uses prior data and experiments for designing future experiments. We illustrate this approach using real data from a ride-sharing platform, yielding a design that reduces MSE by 33% compared to the status quo design used on the platform.

Motivation & Objective

• Address the challenge of designing time-based switchback experiments on a single aggregate unit where treatment effects are confounded by multiple temporal factors.
• Identify and model four key sources of estimation error: carryover effects, periodicity, serially correlated outcomes, and simultaneous interventions.
• Develop a data-driven design framework that improves estimation precision by balancing bias-variance tradeoffs across these factors.
• Demonstrate the practical value of the approach through an empirical application on real ride-sharing platform data.
• Provide a theoretical foundation for bias-variance decomposition that informs optimal interval partitioning and switching frequency.

Proposed method

• Derive a rigorous bias-variance decomposition of the global average treatment effect (GATE) estimation error, isolating contributions from four factors: carryover, periodicity, serial correlation, and simultaneous interventions.
• Propose a novel empirical Bayes design that uses historical experiment data to inform the choice of interval lengths and switching patterns in future experiments.
• Incorporate prior distributions of cumulative effect curves (CECs) and outcome correlations to estimate optimal switching schedules.
• Use a weighted least squares estimation framework to minimize mean squared error (MSE) by balancing bias from carryover effects and variance from correlated outcomes.
• Introduce randomization of interval start and end points to reduce bias and variance from simultaneous experiments.
• Apply the method to real-world data from a ride-sharing platform, using historical treatment effect estimates and outcome dynamics to calibrate the design.

Experimental results

Research questions

• RQ1 How do carryover effects, periodicity, serial correlation, and simultaneous interventions jointly affect the bias and variance of GATE estimation in switchback experiments?
• RQ2 What is the optimal tradeoff between switching frequency (affecting bias from carryover and variance from correlation) in time-based switchback designs?
• RQ3 How can prior experimental data be leveraged to improve the design of future switchback experiments in a data-driven, empirical Bayes framework?
• RQ4 To what extent can randomizing interval boundaries reduce bias and variance in the presence of simultaneous interventions?
• RQ5 What is the empirical gain in estimation precision when applying the proposed design compared to standard status quo designs in real-world platforms?

Key findings

• The proposed empirical Bayes design reduces the mean squared error (MSE) of GATE estimation by 33% compared to the status quo design used on the ride-sharing platform.
• Balancing periodicity between treated and control intervals reduces variance, highlighting the importance of temporal symmetry in experimental design.
• Switching less frequently reduces bias from carryover effects but increases variance from serially correlated outcomes, and vice versa, confirming a clear bias-variance tradeoff.
• Randomizing the start and end points of intervals reduces both bias and variance arising from simultaneous experiments, offering a practical design improvement.
• The theoretical bias-variance decomposition reveals that simultaneous interventions contribute significantly to estimation error, and their impact can be mitigated through strategic interval design.
• The method effectively leverages historical data to calibrate interval lengths and switching patterns, demonstrating strong empirical performance on real-world platform data.