[Paper Review] Take a Fresh Look at Recommender Systems from an Evaluation Standpoint

Aixin Sun|arXiv (Cornell University)|Oct 9, 2022

Recommender Systems and Techniques4 citations

TL;DR

This paper critiques common recommender system evaluation practices, arguing that ignoring the global timeline in train/test splits leads to flawed baselines and counter-intuitive results. It proposes a timeline-aware evaluation scheme to better simulate real-world deployment, improving model transferability and enabling more accurate preference modeling by incorporating decision context over time.

ABSTRACT

Recommendation has become a prominent area of research in the field of Information Retrieval (IR). Evaluation is also a traditional research topic in this community. Motivated by a few counter-intuitive observations reported in recent studies, this perspectives paper takes a fresh look at recommender systems from an evaluation standpoint. Rather than examining metrics like recall, hit rate, or NDCG, or perspectives like novelty and diversity, the key focus here is on how these metrics are calculated when evaluating a recommender algorithm. Specifically, the commonly used train/test data splits and their consequences are re-examined. We begin by examining common data splitting methods, such as random split or leave-one-out, and discuss why the popularity baseline is poorly defined under such splits. We then move on to explore the two implications of neglecting a global timeline during evaluation: data leakage and oversimplification of user preference modeling. Afterwards, we present new perspectives on recommender systems, including techniques for evaluating algorithm performance that more accurately reflect real-world scenarios, and possible approaches to consider decision contexts in user preference modeling.

Motivation & Objective

• To address the disconnect between academic offline evaluation and real-world online deployment in recommender systems.
• To identify how ignoring the global timeline in data splits leads to misleading performance assessments and poor popularity baseline implementation.
• To propose a timeline-aware evaluation scheme that better simulates practical deployment scenarios.
• To encourage modeling user decision-making contexts rather than just user-item interactions for improved preference modeling.
• To bridge the gap between academic research and industrial practice in recommender system evaluation.

Proposed method

• Re-evaluates standard data splitting methods (e.g., random split, leave-one-out) by analyzing their failure to preserve chronological order of user interactions.
• Introduces a timeline scheme for offline evaluation that maintains the global timeline across users, enabling step-by-step evaluation along time.
• Proposes that model training and evaluation should be updated incrementally based on time windows to simulate real-time deployment.
• Advocates for incorporating decision context (e.g., time of interaction, user experience) into preference modeling to reflect real user behavior.
• Recommends retraining models on recent data subsets to align with industrial practices and reduce data leakage risks.
• Suggests that time should be treated as a critical evaluation parameter, not just a feature in model design.

Experimental results

Research questions

• RQ1 Why do highly active users receive poorer recommendations despite having more interaction data?
• RQ2 How does ignoring the global timeline in data splits affect the performance of the popularity baseline?
• RQ3 To what extent does using only recent data improve recommendation performance, and why is this counter-intuitive in standard evaluations?
• RQ4 How can offline evaluation better simulate real-world online deployment conditions in recommender systems?
• RQ5 What role does decision context play in modeling user preferences beyond observed user-item interactions?

Key findings

• Ignoring the global timeline in data splits leads to a poorly defined popularity baseline, undermining the validity of performance comparisons.
• Counter-intuitive results—such as worse recommendations for more active users—arise due to data leakage and improper timeline handling during evaluation.
• Using only recent data subsets significantly improves recommendation performance, indicating that older data may introduce noise or outdated preferences.
• The timeline-aware evaluation scheme reduces data leakage and better simulates real-world deployment, improving model transferability.
• Recommender systems evaluated without timeline awareness produce results that are not indicative of real-world performance.
• Incorporating decision context into preference modeling, informed by timeline-aware evaluation, can lead to more accurate and practical user preference representations.