[Paper Review] Shopping in the Multiverse: A Counterfactual Approach to In-Session Attribution

Jacopo Tagliabue, Bingqing Yu|arXiv (Cornell University)|Jul 20, 2020

Peer-to-Peer Network Technologies25 references4 citations

TL;DR

This paper proposes a causal, counterfactual approach to in-session search attribution in eCommerce by modeling alternative shopper timelines using a generative browsing model over prod2vec-embedded product spaces. It demonstrates that this multiverse-based method (MV) outperforms rule-based (GA), prediction-based (SS), and heuristic methods (CB) in capturing causal relevance, achieving 75% attribution accuracy on an industry dataset, with strong alignment to human intuition on subtle cases.

ABSTRACT

We tackle the challenge of in-session attribution for on-site search engines in eCommerce. We phrase the problem as a causal counterfactual inference, and contrast the approach with rule-based systems from industry settings and prediction models from the multi-touch attribution literature. We approach counterfactuals in analogy with treatments in formal semantics, explicitly modeling possible outcomes through alternative shopper timelines; in particular, we propose to learn a generative browsing model over a target shop, leveraging the latent space induced by prod2vec embeddings; we show how natural language queries can be effectively represented in the same space and how "search intervention" can be performed to assess causal contribution. Finally, we validate the methodology on a synthetic dataset, mimicking important patterns emerged in customer interviews and qualitative analysis, and we present preliminary findings on an industry dataset from a partnering shop.

Motivation & Objective

• To address the limitations of rule-based and prediction-based attribution methods in measuring the true causal impact of on-site search on conversions.
• To model in-session attribution as a counterfactual inference problem, asking whether a conversion would have occurred without the search interaction.
• To develop a practical, minimal-overhead method that leverages dense embeddings and generative modeling to simulate alternative shopper timelines.
• To validate the method on both synthetic and real-world eCommerce datasets, demonstrating improved alignment with human judgment on causal relevance.
• To provide a generalizable framework applicable beyond search to other on-site touchpoints like recommendations and category browsing.

Proposed method

• The method models causal attribution via counterfactual reasoning, treating each shopper session as a potential world where search interactions are hypothetically removed.
• It employs a generative browsing model trained on prod2vec embeddings to simulate alternative timelines, representing products and queries in a shared high-dimensional latent space.
• Natural language queries are embedded into the same space, enabling consistent representation and intervention through 'search intervention' perturbations.
• The model computes the probability of conversion in both observed and counterfactual timelines to estimate causal effect.
• The approach is extended to handle multiple touchpoints by generalizing the multiverse framework across services like recommendations and category listings.
• Validation uses synthetic data mimicking real user behavior and an industry dataset from a partnering eCommerce site, comparing attribution scores across methods.

Experimental results

Research questions

• RQ1 To what extent does on-site search causally contribute to conversion, independent of prior intent or browsing behavior?
• RQ2 How can counterfactual reasoning be effectively applied to high-dimensional, dense representations of user sessions in eCommerce?
• RQ3 Can a generative model of alternative timelines improve attribution accuracy compared to rule-based or prediction-based systems?
• RQ4 How well does the proposed method align with human judgment on subtle attribution cases where search and conversion are loosely connected?
• RQ5 What is the generalizability of the method across different eCommerce platforms and touchpoints beyond search?

Key findings

• The multiverse-based method (MV) achieved 75% search attribution on an industry dataset, outperforming rule-based (GA: 100%, but misleading), causal baseline (CB: 47%), and similarity-based (SS: 77%) methods.
• MV correctly identified cases where search had minimal causal impact—such as when a user searched after already adding the target product to cart—marking such interactions as weakly relevant, unlike SS which overestimated relevance.
• On synthetic data, MV successfully captured nuanced causal dynamics, such as when search led to browsing but not conversion, or when conversion occurred independently of search.
• The method demonstrated strong qualitative alignment with human intuition, particularly in cases where prior intent and search relevance were decoupled.
• The framework is generalizable and extensible, with straightforward adaptation to other on-site features like recommendations and category navigation.
• The study confirms that relevance to underlying shopper intent is a crucial factor in accurate causal attribution, and that purely predictive models (like SS) fail to capture this nuance.