[Paper Review] Estimating Network Effects Using Naturally Occurring Peer Notification Queue Counterfactuals

Craig Tutterow, Guillaume Saint-Jacques|arXiv (Cornell University)|Feb 19, 2019

Advanced Causal Inference Techniques21 references20 citations

TL;DR

This paper proposes using naturally occurring peer notification queue ordering in online platforms like LinkedIn as a source of exogenous variation to estimate network effects without experimental intervention. By exploiting the random-like ordering of messages in scalable systems, the authors identify two natural experiments that reveal significant peer influence on user engagement, showing that standard fixed-effects models can overestimate these effects by up to 2.7x.

ABSTRACT

Randomized experiments, or A/B tests are used to estimate the causal impact of a feature on the behavior of users by creating two parallel universes in which members are simultaneously assigned to treatment and control. However, in social network settings, members interact, such that the impact of a feature is not always contained within the treatment group. Researchers have developed a number of experimental designs to estimate network effects in social settings. Alternatively, naturally occurring exogenous variation, or 'natural experiments,' allow researchers to recover causal estimates of peer effects from observational data in the absence of experimental manipulation. Natural experiments trade off the engineering costs and some of the ethical concerns associated with network randomization with the search costs of finding situations with natural exogenous variation. To mitigate the search costs associated with discovering natural counterfactuals, we identify a common engineering requirement used to scale massive online systems, in which natural exogenous variation is likely to exist: notification queueing. We identify two natural experiments on the LinkedIn platform based on the order of notification queues to estimate the causal impact of a received message on the engagement of a recipient. We show that receiving a message from another member significantly increases a member's engagement, but that some popular observational specifications, such as fixed-effects estimators, overestimate this effect by as much as 2.7x. We then apply the estimated network effect coefficients to a large body of past experiments to quantify the extent to which it changes our interpretation of experimental results. The study points to the benefits of using messaging queues to discover naturally occurring counterfactuals for the estimation of causal effects without experimenter intervention.

Motivation & Objective

• To estimate peer network effects in social platforms without relying on randomized A/B tests.
• To reduce the cost and ethical concerns of network randomization by leveraging naturally occurring exogenous variation.
• To identify scalable engineering artifacts—specifically notification queue ordering—as sources of valid counterfactuals.
• To quantify the bias introduced by conventional observational models like fixed-effects estimators in network effect estimation.
• To demonstrate the practical utility of notification queues as a discovery mechanism for natural experiments in large-scale systems.

Proposed method

• Leverages the inherent randomness in notification queue ordering within massive online systems as a source of exogenous variation.
• Identifies two natural experiments on LinkedIn where the order of message delivery creates quasi-random assignment to treatment and control conditions.
• Uses the timing of message receipt relative to peers' messages as a proxy for causal exposure, enabling causal inference.
• Applies structural econometric models to estimate peer effect coefficients while controlling for unobserved heterogeneity.
• Compares results from natural experiment-based estimation with those from standard fixed-effects models to assess bias.
• Validates findings by applying estimated network effects to historical experimental data to re-evaluate prior results.

Experimental results

Research questions

• RQ1 Can naturally occurring variation in notification queue ordering serve as a valid source of exogenous variation for estimating peer effects?
• RQ2 How do standard observational models like fixed-effects estimators perform when estimating network effects in the presence of peer influence?
• RQ3 What is the magnitude of bias introduced by conventional modeling approaches in network effect estimation?
• RQ4 To what extent do network effects alter the interpretation of prior A/B test results in social platforms?
• RQ5 Can engineering artifacts like notification queues be systematically exploited to discover natural experiments at scale?

Key findings

• Receiving a message from a peer significantly increases a user’s subsequent engagement on the platform.
• Fixed-effects estimators overestimate the causal impact of peer messages by up to 2.7 times compared to estimates derived from natural experiment counterfactuals.
• Notification queue ordering provides a reliable source of exogenous variation that enables valid causal inference without experimental manipulation.
• The study demonstrates that network effects are systematically underestimated or distorted when using standard observational models in social network settings.
• Applying estimated network effect coefficients to past experiments reveals that prior interpretations of A/B test results may be substantially biased.
• The approach offers a scalable, low-cost alternative to randomized experiments for estimating peer influence in large-scale online systems.