[Paper Review] Open Benchmarking for Click-Through Rate Prediction

Jieming Zhu, Jinyang Liu+3 more|arXiv (Cornell University)|Sep 12, 2020

Green IT and Sustainability49 references4 citations

TL;DR

This paper introduces BARS-CTR, the first open benchmark for click-through rate (CTR) prediction, standardizing evaluation protocols across 24 models on Criteo and Avazu datasets. After over 7,000 experiments and 12,000 GPU hours, it reveals that model performance differences are smaller than expected after thorough hyperparameter tuning, highlighting the need for reproducible benchmarking in CTR research.

ABSTRACT

Click-through rate (CTR) prediction is a critical task for many applications, as its accuracy has a direct impact on user experience and platform revenue. In recent years, CTR prediction has been widely studied in both academia and industry, resulting in a wide variety of CTR prediction models. Unfortunately, there is still a lack of standardized benchmarks and uniform evaluation protocols for CTR prediction research. This leads to non-reproducible or even inconsistent experimental results among existing studies, which largely limits the practical value and potential impact of their research. In this work, we build an open benchmark for CTR prediction, namely BARS-CTR, and present a rigorous comparison of different models in a reproducible manner. To this end, we ran over 7,000 experiments for more than 12,000 GPU hours in total to re-evaluate 24 existing models on multiple datasets and settings. Surprisingly, our experiments show that with sufficient hyper-parameter search and model tuning, many deep models have smaller differences than expected. The results also reveal that making real progress on the modeling of CTR prediction is indeed a very challenging research task. We believe that our benchmarking work could not only allow researchers to gauge the effectiveness of new models conveniently but also make them fairly compare with the state of the arts. We have publicly released the benchmarking code, evaluation protocols, and hyper-parameter settings of our work to promote reproducible research in this field.

Motivation & Objective

• Address the lack of standardized evaluation protocols and reproducible results in CTR prediction research.
• Provide a unified benchmarking framework to enable fair and consistent comparison across existing CTR models.
• Reduce redundant re-implementation of baselines by researchers by releasing full training configurations and results.
• Highlight inconsistencies and non-reproducibility in prior studies due to varying data splits, preprocessing, and hyperparameter settings.
• Promote reproducible research by publicly releasing code, hyperparameters, and intermediate artifacts for all experiments.

Proposed method

• Built BARS-CTR, an open benchmark platform for CTR prediction, standardizing data preprocessing, train-test splits, and evaluation metrics.
• Re-evaluated 24 existing CTR models using a uniform training setup across multiple runs with fixed random seeds.
• Performed extensive hyperparameter search using Bayesian optimization for each model and dataset combination.
• Used two widely adopted real-world datasets—Criteo and Avazu—under multiple data partitioning schemes to ensure robustness.
• Standardized preprocessing steps including handling of numerical features, rare categorical features, and feature field definitions.
• Released full training logs, hyperparameter configurations, and code to ensure full reproducibility of all results.

Experimental results

Research questions

• RQ1 To what extent do existing CTR models differ in performance when evaluated under a standardized, reproducible protocol?
• RQ2 How much do hyperparameter tuning and consistent data preprocessing affect the relative ranking of CTR models?
• RQ3 Why do many prior studies report inconsistent or non-reproducible results despite using widely available datasets like Criteo and Avazu?
• RQ4 Can a centralized, open benchmark improve the reliability and fairness of model comparisons in CTR prediction research?
• RQ5 What is the true performance gap between state-of-the-art CTR models after rigorous tuning and standardized evaluation?

Key findings

• After extensive hyperparameter tuning and standardized evaluation, the performance differences between many deep CTR models are significantly smaller than previously reported.
• The benchmark reveals that inconsistent data splits, preprocessing methods, and hyperparameter choices in prior work lead to non-reproducible and misleading comparisons.
• Even models considered state-of-the-art in prior studies show only marginal improvements over strong baselines when evaluated under the same protocol.
• The results demonstrate that achieving meaningful accuracy gains in CTR prediction remains a highly challenging task, even with advanced deep learning architectures.
• The open release of training logs, hyperparameters, and code enables full reproducibility and allows researchers to validate and compare models fairly.
• The benchmarking effort consumed over 12,000 GPU hours and involved more than 7,000 individual experiments, underscoring the complexity of fair model evaluation in CTR prediction.