[Paper Review] Markovian Interference in Experiments

Vivek F. Farias, Andrew A. Li+2 more|arXiv (Cornell University)|Jun 6, 2022

Transportation and Mobility Innovations18 citations

TL;DR

This paper introduces the Differences-In-Q's (DQ) estimator for causal inference in experiments with Markovian interference, where interventions affect system states due to shared constraints like limited inventory. By leveraging a novel on-policy estimation framework with second-order bias and exponentially reduced variance compared to off-policy methods, DQ achieves superior bias-variance tradeoffs, outperforming both heuristic and unbiased estimators in theory and simulation.

ABSTRACT

We consider experiments in dynamical systems where interventions on some experimental units impact other units through a limiting constraint (such as a limited inventory). Despite outsize practical importance, the best estimators for this `Markovian' interference problem are largely heuristic in nature, and their bias is not well understood. We formalize the problem of inference in such experiments as one of policy evaluation. Off-policy estimators, while unbiased, apparently incur a large penalty in variance relative to state-of-the-art heuristics. We introduce an on-policy estimator: the Differences-In-Q's (DQ) estimator. We show that the DQ estimator can in general have exponentially smaller variance than off-policy evaluation. At the same time, its bias is second order in the impact of the intervention. This yields a striking bias-variance tradeoff so that the DQ estimator effectively dominates state-of-the-art alternatives. From a theoretical perspective, we introduce three separate novel techniques that are of independent interest in the theory of Reinforcement Learning (RL). Our empirical evaluation includes a set of experiments on a city-scale ride-hailing simulator.

Motivation & Objective

• To address the lack of theoretically grounded, low-bias, low-variance estimators for average treatment effect (ATE) in experiments with Markovian interference, where interventions affect shared system states.
• To formalize the problem as policy evaluation in a Markov Decision Process (MDP), particularly under simple randomization, avoiding reliance on complex experimental designs.
• To develop an on-policy estimator that achieves provably smaller variance than off-policy evaluation while maintaining second-order bias in the treatment effect.
• To establish theoretical foundations in reinforcement learning, including a novel Taylor-like expansion of ATE and bias-variance analysis for policy evaluation.

Proposed method

• Proposes the DQ estimator as a novel on-policy estimator that uses a modified form of the policy evaluation identity, replacing standard rewards with expected rewards under the target policy.
• Derives a surrogate objective based on the stationary distribution and value functions of a behavior policy, using perturbation bounds to analyze bias.
• Employs a Taylor-like expansion of the ATE (Theorem 5) to characterize bias in terms of the distance between transition probabilities and policies.
• Introduces a modified Dynkin-type identity for policy evaluation, where the Q- and V-functions are computed on an auxiliary MDP with rewards reweighted by the target policy’s expected reward.
• Uses a trust-region-like framework but modifies the reward function in the auxiliary MDP to reduce bias, leading to the DQ estimator’s key properties.
• Establishes asymptotic normality and provides explicit variance characterization for the DQ estimator, showing it can be exponentially smaller than off-policy estimators.

Experimental results

Research questions

• RQ1 Can a novel on-policy estimator achieve significantly lower variance than unbiased off-policy estimators in experiments with Markovian interference?
• RQ2 What is the bias structure of the DQ estimator, and how does it compare to existing off-policy and heuristic estimators?
• RQ3 Can the DQ estimator be derived from a modified policy evaluation identity that accounts for policy-induced changes in transition dynamics?
• RQ4 Does the DQ estimator maintain second-order bias when the treatment effect is small, and how does this affect its performance in practice?
• RQ5 Can the DQ estimator be generalized to improve policy optimization in reinforcement learning through better bias-variance control?

Key findings

• The DQ estimator exhibits second-order bias, $O(\delta^2)$, when the intervention's impact on transition probabilities is $O(\delta)$, significantly reducing bias compared to off-policy methods.
• The variance of the DQ estimator can be exponentially smaller than that of any unbiased off-policy estimator, particularly in high-dimensional or complex MDPs.
• In a city-scale ride-hailing simulator, the DQ estimator outperformed state-of-the-art alternatives in both bias and mean squared error, demonstrating practical superiority.
• The DQ estimator's bias is $O((\delta\delta')^2)$, which is a factor $\delta$ smaller than the $O(\delta(\delta')^2)$ bias of the trust-region surrogate, even when $\delta'$ is unbounded.
• Theoretical analysis reveals that the DQ estimator leverages the smallness of treatment effects to achieve a favorable bias-variance tradeoff, making it dominant in both theory and simulation.
• The method introduces three novel techniques of independent interest in RL: a Taylor-like expansion of ATE, a modified Dynkin identity, and a perturbation-based analysis of policy evaluation bias.