[Paper Review] BARS: Towards Open Benchmarking for Recommender Systems

Jieming Zhu, Quanyu Dai+6 more|arXiv (Cornell University)|May 19, 2022

Recommender Systems and Techniques10 citations

TL;DR

BARS introduces a standardized, open benchmarking pipeline for recommender systems that unifies datasets, code, hyperparameters, logs, and evaluation results to enable reproducible research. It reveals that many claimed performance gains in recent CTR prediction models are exaggerated, as top models achieve near-identical AUC scores (e.g., ~81.4 on Criteo_x4), underscoring the need for rigorous, consistent evaluation.

ABSTRACT

The past two decades have witnessed the rapid development of personalized recommendation techniques. Despite significant progress made in both research and practice of recommender systems, to date, there is a lack of a widely-recognized benchmarking standard in this field. Many existing studies perform model evaluations and comparisons in an ad-hoc manner, for example, by employing their own private data splits or using different experimental settings. Such conventions not only increase the difficulty in reproducing existing studies, but also lead to inconsistent experimental results among them. This largely limits the credibility and practical value of research results in this field. To tackle these issues, we present an initiative project (namely BARS) aiming for open benchmarking for recommender systems. In comparison to some earlier attempts towards this goal, we take a further step by setting up a standardized benchmarking pipeline for reproducible research, which integrates all the details about datasets, source code, hyper-parameter settings, running logs, and evaluation results. The benchmark is designed with comprehensiveness and sustainability in mind. It covers both matching and ranking tasks, and also enables researchers to easily follow and contribute to the research in this field. This project will not only reduce the redundant efforts of researchers to re-implement or re-run existing baselines, but also drive more solid and reproducible research on recommender systems. We would like to call upon everyone to use the BARS benchmark for future evaluation, and contribute to the project through the portal at: https://openbenchmark.github.io/BARS.

Motivation & Objective

• Address the lack of standardized, reproducible evaluation in recommender systems research.
• Reduce ad-hoc evaluation practices such as private data splits and inconsistent baselines.
• Establish a unified pipeline integrating datasets, code, hyperparameters, logs, and results for transparency.
• Promote fair comparisons and reduce exaggerated claims of model improvements.
• Encourage community contributions to sustain and expand the benchmark over time.

Proposed method

• Design a standardized benchmarking pipeline that includes datasets, source code, hyperparameter settings, training logs, and evaluation metrics.
• Integrate multiple real-world datasets, including Criteo, Avazu, MovieLens, and AmazonBooks, with standardized preprocessing and splits.
• Support both matching (retrieval) and ranking (CTR prediction) tasks in a unified framework.
• Enable reproducibility by publishing complete training configurations and results across multiple random seeds.
• Host the benchmark on a public portal (https://openbenchmark.github.io/BARS) for community access and contribution.
• Provide tools and guidelines for researchers to reproduce, tune, and validate models consistently.

Experimental results

Research questions

• RQ1 To what extent do existing recommender system evaluations lack reproducibility due to inconsistent data splits and hyperparameter settings?
• RQ2 How do state-of-the-art CTR prediction models compare when evaluated under identical, standardized conditions?
• RQ3 Are the performance improvements reported in recent papers truly significant, or are they inflated due to poor baseline tuning or experimental design?
• RQ4 Can a unified, open benchmarking pipeline reduce redundant implementation efforts and improve research credibility?
• RQ5 What are the key components needed to sustain a long-term, community-driven benchmark for recommender systems?

Key findings

• DeepFM, DCN, xDeepFM, and DCN-v2 achieve nearly identical AUC scores (~81.4) on the Criteo_x4 dataset, indicating minimal performance differences despite claimed improvements.
• PNN, HFM+, and FGCNN achieve similar AUC scores (~79.44) on Avazu_x4, suggesting limited gains from newer architectures under standardized settings.
• Several recently proposed models, including InterHAt, AFN+, and LorentzFM, underperform compared to established baselines, indicating potential overstatement of improvements in original papers.
• The observed performance differences among top models are often less than 0.1%, highlighting the difficulty of achieving significant gains in large-scale CTR prediction tasks.
• Many reported results in the literature are not reproducible due to missing details on data splits, hyperparameters, or implementation, undermining credibility.
• The BARS benchmark enables consistent, repeatable evaluation and reveals inconsistencies in prior work, supporting the need for standardized, open benchmarking.