[Paper Review] A/B Testing for Recommender Systems in a Two-sided Marketplace

Preetam Nandy, Divya Venugopalan+2 more|arXiv (Cornell University)|May 29, 2021

Advanced Causal Inference Techniques18 references9 citations

TL;DR

This paper proposes UniCoRn, a novel A/B testing framework for two-sided marketplaces that enables high-quality, unbiased measurement of treatment effects on producers by unifying counterfactual rankings across conflicting treatment assignments. It achieves optimal experiment design with explicit control over accuracy and computational cost, without requiring prior knowledge of the network structure or assumptions about treatment propagation.

ABSTRACT

Two-sided marketplaces are standard business models of many online platforms (e.g., Amazon, Facebook, LinkedIn), wherein the platforms have consumers, buyers or content viewers on one side and producers, sellers or content-creators on the other. Consumer side measurement of the impact of a treatment variant can be done via simple online A/B testing. Producer side measurement is more challenging because the producer experience depends on the treatment assignment of the consumers. Existing approaches for producer side measurement are either based on graph cluster-based randomization or on certain treatment propagation assumptions. The former approach results in low-powered experiments as the producer-consumer network density increases and the latter approach lacks a strict notion of error control. In this paper, we propose (i) a quantification of the quality of a producer side experiment design, and (ii) a new experiment design mechanism that generates high-quality experiments based on this quantification. Our approach, called UniCoRn (Unifying Counterfactual Rankings), provides explicit control over the quality of the experiment and its computation cost. Further, we prove that our experiment design is optimal to the proposed design quality measure. Our approach is agnostic to the density of the producer-consumer network and does not rely on any treatment propagation assumption. Moreover, unlike the existing approaches, we do not need to know the underlying network in advance, making this widely applicable to the industrial setting where the underlying network is unknown and challenging to predict a priori due to its dynamic nature. We use simulations to validate our approach and compare it against existing methods. We also deployed UniCoRn in an edge recommendation application that serves tens of millions of members and billions of edge recommendations daily.

Motivation & Objective

• To address the challenge of measuring treatment effects on producers in two-sided marketplaces where SUTVA is violated due to network dependencies.
• To develop an experiment design mechanism that ensures high accuracy and low variance in producer-side measurement without relying on graph density or treatment propagation assumptions.
• To provide explicit control over the trade-off between experiment quality and computational cost.
• To design a method that works in dynamic, real-world environments where the underlying producer-consumer network is unknown and constantly changing.
• To prove the optimality of the proposed design under a newly defined inaccuracy measure for experiment quality.

Proposed method

• The method introduces a design quality measure based on the inaccuracy of exposure distribution estimation between treatment and control groups.
• It formulates a unified counterfactual ranking system that resolves conflicts arising when producers are assigned to different treatment variants.
• The approach uses a randomized treatment allocation strategy on the producer side, with a unique ranking per treatment variant, and unifies these into a single consistent ranking framework.
• It enables the use of larger treatment ramps (e.g., higher producer-side treatment groups) without increasing variance, improving statistical power.
• The design is provably optimal with respect to the defined inaccuracy measure, ensuring minimal estimation error.
• The framework does not require prior knowledge of the network structure, making it suitable for dynamic, real-time marketplace applications.

Experimental results

Research questions

• RQ1 How can we design A/B experiments in two-sided marketplaces that accurately measure treatment effects on producers when SUTVA is violated?
• RQ2 What is a principled, measurable way to quantify the quality of a producer-side experiment design?
• RQ3 Can we design an experiment mechanism that is optimal with respect to this quality measure and independent of network density or treatment propagation assumptions?
• RQ4 How can we balance the trade-off between experiment accuracy and computational cost in large-scale, dynamic marketplaces?
• RQ5 Can the proposed method be applied to real-world systems without requiring prior knowledge of the underlying network structure?

Key findings

• UniCoRn achieves optimal experiment design with respect to the proposed inaccuracy measure, ensuring minimal estimation error in exposure distribution.
• The method enables higher statistical power by supporting larger treatment ramps on the producer side without increasing variance.
• Simulations and real-world deployment at LinkedIn demonstrated positive impacts on WAUs and sessions, indicating a statistically significant treatment effect.
• The framework is agnostic to graph density and does not require assumptions about treatment propagation or response dependence on exposure.
• The approach does not require prior knowledge of the network structure, making it suitable for dynamic, real-world marketplace applications.
• The method outperforms existing approaches in terms of measurement accuracy and robustness, especially in dense or evolving networks.