[Paper Review] Exploring Data Splitting Strategies for the Evaluation of Recommendation Models

Zaiqiao Meng, Richard McCreadie+2 more|arXiv (Cornell University)|Jul 26, 2020

Recommender Systems and Techniques26 references4 citations

TL;DR

This paper investigates how different data splitting strategies significantly impact the evaluation and ranking of recommendation models, demonstrating that the same model can rank differently across strategies—even when using the same dataset and metrics. The authors show that strategies like leave-one-last and temporal user splits can leak future information, leading to unreliable comparisons, and advocate for standardized temporal global splitting as the default for fair evaluation.

ABSTRACT

Effective methodologies for evaluating recommender systems are critical, so that such systems can be compared in a sound manner. A commonly overlooked aspect of recommender system evaluation is the selection of the data splitting strategy. In this paper, we both show that there is no standard splitting strategy and that the selection of splitting strategy can have a strong impact on the ranking of recommender systems. In particular, we perform experiments comparing three common splitting strategies, examining their impact over seven state-of-the-art recommendation models for two datasets. Our results demonstrate that the splitting strategy employed is an important confounding variable that can markedly alter the ranking of state-of-the-art systems, making much of the currently published literature non-comparable, even when the same dataset and metrics are used.

Motivation & Objective

• To investigate the impact of data splitting strategies on the performance and ranking of recommendation models.
• To identify inconsistencies in current evaluation practices across the literature, particularly regarding data split methodologies.
• To demonstrate that different splitting strategies can lead to markedly different rankings of state-of-the-art models, even under identical datasets and metrics.
• To provide best practices for future research, including reporting and sharing data splits.

Proposed method

• The authors evaluate seven state-of-the-art recommendation models—BPR, NMF, VAECF, NGCF, NeuMF, VBCAR, and Triple2Vec—across three data splitting strategies: leave-one-item, leave-one-basket, and temporal global.
• They apply these strategies to two real-world datasets: Amazon and Dunnhumby, ensuring consistent preprocessing and hyperparameter tuning.
• Performance is measured using standard metrics: NDCG@10 and Recall@10 across all splits.
• A large-scale correlation experiment is conducted using 230 model variants generated by hyperparameter tuning of three core models (NeuMF, VBCAR, Triple2Vec) to assess ranking stability across strategies.
• Kendall’s τ correlation is computed between score distributions of each pair of splitting strategies to quantify ranking consistency.
• The authors analyze leakage in leave-one-last and temporal user splits, identifying that future interactions can inadvertently inform training, distorting evaluation.

Experimental results

Research questions

• RQ1 Does the choice of data splitting strategy significantly alter the ranking of state-of-the-art recommendation models?
• RQ2 To what extent do different splitting strategies lead to inconsistent performance rankings when using the same dataset and metrics?
• RQ3 Which splitting strategy minimizes information leakage and provides the most realistic evaluation of model performance?
• RQ4 Are certain recommendation models systematically favored by specific splitting strategies due to data distribution or leakage?
• RQ5 What best practices should be adopted to ensure reproducibility and fairness in future recommendation system evaluations?

Key findings

• The temporal global splitting strategy produces the most consistent and realistic evaluation, and should be used as the default for fair comparison.
• Leave-one-last and temporal user splits exhibit significant information leakage, where future interactions are inadvertently available during training, distorting performance assessment.
• Kendall’s τ correlation between splitting strategies ranges from 0.5284 to 0.7630, indicating substantial rank swaps across strategies, even among high-performing models.
• The leave-one-item strategy produces wider score dispersion in NDCG@10 and Recall@10, suggesting it evaluates different aspects of model behavior compared to more structured splits.
• Certain models, such as Triple2Vec and VBCAR, show stronger performance under specific splits—indicating that model ranking is sensitive to data split design.
• The study confirms that data splitting strategy is a critical confounding variable in model evaluation, undermining comparability across published works despite identical datasets and metrics.