[Paper Review] Study of a bias in the offline evaluation of a recommendation algorithm

Arnaud De Myttenaere, Boris Golden+2 more|arXiv (Cornell University)|Nov 4, 2015

Recommender Systems and Techniques3 citations

TL;DR

This paper identifies a bias in offline evaluation of recommendation algorithms caused by prior recommendations influencing historical data. It proposes a weighted offline evaluation method using instance weighting to correct for this bias, demonstrating significant reduction in evaluation error—especially for constant and collaborative filtering algorithms—on real-world Viadeo data, though structural changes from campaigns limit full stabilization of scores.

ABSTRACT

Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical data (via offline evaluation). This paper describes this bias and discuss the relevance of a weighted offline evaluation to reduce this bias for different classes of recommendation algorithms.

Motivation & Objective

• To identify and analyze a systematic bias in offline evaluation of recommendation algorithms caused by prior recommendations influencing historical data.
• To evaluate the effectiveness of instance weighting as a correction method for this bias in offline evaluation.
• To assess whether weighted evaluation improves the accuracy of performance estimation for both simple (constant) and complex (collaborative filtering) recommendation algorithms.
• To investigate the dual nature of the bias: one due to changed item selection probabilities, and another due to structural changes in user-item interaction graphs.
• To demonstrate the practical relevance of the weighting approach on real-world data from Viadeo’s professional social network.

Proposed method

• The paper introduces a weighted offline evaluation framework that assigns instance-specific weights to user-item pairs based on their likelihood under a reference distribution, correcting for covariate shift.
• It formulates the evaluation as a weighted average of prediction accuracy, where weights are optimized to minimize the discrepancy between training and evaluation data distributions.
• The method uses a probabilistic model to estimate the probability of each item being selected for evaluation, adjusting for the influence of past recommendation campaigns.
• For collaborative filtering, the method applies the weighting scheme to both cosine similarity and naive CF algorithms, using the same optimization procedure to adjust for bias.
• The weights are computed iteratively over time, with the number of optimized weights (p) controlling the degree of recalibration.
• The approach is validated on a 201-day period from Viadeo data, including two major recommendation campaigns at t=330 and t=430.

Experimental results

Research questions

• RQ1 How does the presence of prior recommendations in historical data bias offline evaluation of recommendation algorithms?
• RQ2 To what extent can instance weighting reduce the bias in offline evaluation for constant recommendation algorithms?
• RQ3 Can the same weighting strategy effectively reduce bias in more complex collaborative filtering algorithms?
• RQ4 What are the relative contributions of selection probability shifts and structural changes in the user-item graph to the overall evaluation bias?
• RQ5 How does the number of optimized weights (p) affect the performance and stability of the corrected evaluation?

Key findings

• Offline evaluation scores strongly decreased after Viadeo’s two recommendation campaigns at t=330 and t=430, indicating significant bias from prior recommendations.
• The proposed weighting method effectively reduced the bias in offline evaluation, with higher values of p (more optimized weights) leading to greater correction.
• For constant recommendation algorithms, the weighted evaluation stabilized scores after campaigns, demonstrating strong correction of selection bias.
• For collaborative filtering algorithms, the method reduced the selection-probability-based bias but could not fully stabilize scores due to structural changes in the user-item graph.
• The bias was found to have two components: one from altered item selection probabilities and another from increased graph density due to campaign-driven user-item connections.
• The results confirm that instance weighting is a practical and effective approach to reduce evaluation bias in offline settings, especially when random data collection is infeasible.