[Paper Review] Recommendations as Treatments: Debiasing Learning and Evaluation

Tobias Schnabel, Adith Swaminathan+3 more|arXiv (Cornell University)|Feb 17, 2016

Advanced Bandit Algorithms Research45 references143 citations

TL;DR

The paper treats recommendations as causal interventions to debias learning and evaluation under MNAR data, proposing propensity-weighted estimators and a scalable matrix factorization approach with theoretical guarantees and robust propensity estimation.

ABSTRACT

Most data for evaluating and training recommender systems is subject to selection biases, either through self-selection by the users or through the actions of the recommendation system itself. In this paper, we provide a principled approach to handling selection biases, adapting models and estimation techniques from causal inference. The approach leads to unbiased performance estimators despite biased data, and to a matrix factorization method that provides substantially improved prediction performance on real-world data. We theoretically and empirically characterize the robustness of the approach, finding that it is highly practical and scalable.

Motivation & Objective

• Address selection biases in recommender system data by framing recommendations as interventions in causal inference.
• Develop unbiased performance estimators for common metrics under MNAR conditions (e.g., MAE, MSE, DCG).
• Propose an ERM framework for learning with biases and derive a scalable matrix factorization method.
• Investigate propensity estimation methods and robustness under mis-specified propensities.

Proposed method

• Use propensity-weighting (IPS) and complete-case ideas to create unbiased estimators for MAE, MSE, DCG, and related metrics.
• Formulate an Empirical Risk Minimization (ERM) objective under propensities and derive a generalization bound.
• Derive a propensity-scored matrix factorization objective that resembles standard incomplete MF but with per-entry weights 1/P_{u,i}.
• Introduce SNIPS to reduce variance of IPS by self-normalization, trading a small bias for lower variance.
• Distinguish experimental (known P) and observational (estimated P) settings and analyze robustness to propensity mis-specification.
• Provide methods to estimate propensities via Naive Bayes and logistic regression (with user/item biases).

Experimental results

Research questions

• RQ1 How do selection biases distort evaluation and learning in recommender systems, and can causal-inference techniques yield unbiased estimators?
• RQ2 Can propensity-based estimators enable unbiased evaluation across common metrics (MAE, MSE, DCG) under MNAR data?
• RQ3 Can we develop a scalable matrix factorization method that accounts for selection bias while maintaining performance?
• RQ4 How do propensities need to be estimated in observational settings, and how robust is the approach to mis-specification?

Key findings

• IPS and SNIPS provide unbiased (IPS) or low-variance unbiased (SNIPS) estimates of MAE and DCG compared to naive, which are biased under MNAR.
• Propensity-scored MF (MF-IPS) improves rating prediction accuracy (lower MSE) over unweighted MF under varying MNAR severity in semi-synthetic data.
• Empirical risk minimization with propensity weighting yields generalization guarantees for finite hypothesis spaces.
• Estimated propensities (even when imperfect) still yield unbiased or robust evaluation and learning performance, often outperforming naive approaches.
• Compared to state-of-the-art joint-likelihood MNAR methods, the proposed approach is more scalable and competitive in real-world datasets (Yahoo! R3 and Coat Shopping).
• The paper provides an implementation and a benchmark dataset for MNAR evaluation.