[Paper Review] Counterfactual inference in sequential experiments

Raaz Dwivedi, Katherine Tian+4 more|arXiv (Cornell University)|Feb 14, 2022

Statistical Methods in Clinical Trials4 citations

TL;DR

This paper proposes a non-parametric latent factor model for counterfactual inference in sequentially designed experiments with adaptive treatment policies, using a nearest-neighbor-based estimation method to achieve valid confidence intervals for unit- and time-specific counterfactual means. Under regularity conditions, the method provides asymptotically valid inference with high probability error bounds, even under minimal assumptions on the adaptive policy.

ABSTRACT

We consider after-study statistical inference for sequentially designed experiments wherein multiple units are assigned treatments for multiple time points using treatment policies that adapt over time. Our goal is to provide inference guarantees for the counterfactual mean at the smallest possible scale -- mean outcome under different treatments for each unit and each time -- with minimal assumptions on the adaptive treatment policy. Without any structural assumptions on the counterfactual means, this challenging task is infeasible due to more unknowns than observed data points. To make progress, we introduce a latent factor model over the counterfactual means that serves as a non-parametric generalization of the non-linear mixed effects model and the bilinear latent factor model considered in prior works. For estimation, we use a non-parametric method, namely a variant of nearest neighbors, and establish a non-asymptotic high probability error bound for the counterfactual mean for each unit and each time. Under regularity conditions, this bound leads to asymptotically valid confidence intervals for the counterfactual mean as the number of units and time points grows to $\infty$ together at suitable rates. We illustrate our theory via several simulations and a case study involving data from a mobile health clinical trial HeartSteps.

Motivation & Objective

• To enable after-study statistical inference for counterfactual means at the unit × time level in sequentially designed experiments with adaptive treatment policies.
• To address the challenge of high-dimensional, underdetermined counterfactual estimation when the number of unknowns exceeds observed data points.
• To develop a method that requires minimal structural assumptions on counterfactual means while ensuring valid inference under adaptive, potentially pooled, treatment policies.
• To establish non-asymptotic high-probability error bounds for counterfactual mean estimates and derive asymptotically valid confidence intervals as N and T grow.
• To validate the method empirically through simulations and a real-world mobile health trial (HeartSteps), demonstrating robustness to user heterogeneity and policy adaptivity.

Proposed method

• Introduces a non-parametric latent factor model over counterfactual means, generalizing non-linear mixed effects and bilinear factor models.
• Uses a variant of the nearest neighbors algorithm to estimate counterfactual means, leveraging similarity across units in treatment-response space.
• Derives a non-asymptotic high-probability error bound for the estimated counterfactual mean per unit and time point.
• Establishes asymptotic validity of confidence intervals by showing convergence rates under joint growth of N and T.
• Employs a data-driven hyperparameter tuning strategy for the nearest-neighbor threshold η using training data, avoiding validation sets due to limited data.
• Imputes missing outcomes using treatment- and user-specific means for available times, and explores alternative imputation strategies (e.g., 'available' and 'zero' imputation) for non-available times in sensitivity analysis.

Experimental results

Research questions

• RQ1 Can valid confidence intervals be constructed for unit- and time-specific counterfactual means in sequential experiments with adaptive, pooled treatment policies?
• RQ2 How can counterfactual inference be achieved at the finest scale (unit × time) when the number of unknowns exceeds the number of observed data points?
• RQ3 To what extent does the proposed non-parametric latent factor model capture heterogeneity in treatment effects across units and time without strong parametric assumptions?
• RQ4 How does the nearest-neighbor estimation strategy perform in terms of empirical coverage and robustness to user heterogeneity in real-world data?
• RQ5 What is the impact of different imputation strategies for missing outcomes on the singular value decomposition of the outcome matrix and downstream inference?

Key findings

• The proposed method achieves non-asymptotic high-probability error bounds for counterfactual mean estimation, enabling valid inference under minimal assumptions.
• As N and T grow jointly at suitable rates, the confidence intervals for counterfactual means are asymptotically valid, ensuring correct coverage in large samples.
• The tuned hyperparameter η = 8.72 (43rd percentile of pairwise distances) yields at least three neighbors for all 35 filtered users in the HeartSteps case study, ensuring stable estimation.
• A negative correlation (r = -0.69, p = 0.06) is observed between user-specific random effects from IP-TS and the number of nearest neighbors, suggesting alignment in capturing user heterogeneity.
• The 'available' imputation strategy explains over 60% of variance with the first three singular components, while the 'zero' imputation strategy requires around ten components for the same variance explanation.
• Empirical coverage across 35 filtered users shows that the method maintains close to nominal coverage levels, with a histogram indicating most users achieve coverage near the target.