[Paper Review] GRecX: An Efficient and Unified Benchmark for GNN-based Recommendation

Desheng Cai, Jun Hu+4 more|arXiv (Cornell University)|Nov 19, 2021

Recommender Systems and Techniques17 references4 citations

TL;DR

GRecX is an open-source TensorFlow framework that enables efficient and unified benchmarking of GNN-based recommendation models by providing optimized libraries for metrics, similarity search, mini-batch evaluation, and dataset management. It introduces a new metric, GRMF-X, and offers highly efficient, reproducible implementations of popular models like NGCF, LightGCN, and UltraGCN, significantly improving inference speed and enabling fair comparisons across models with varying hyperparameters.

ABSTRACT

In this paper, we present GRecX, an open-source TensorFlow framework for benchmarking GNN-based recommendation models in an efficient and unified way. GRecX consists of core libraries for building GNN-based recommendation benchmarks, as well as the implementations of popular GNN-based recommendation models. The core libraries provide essential components for building efficient and unified benchmarks, including FastMetrics (efficient metrics computation libraries), VectorSearch (efficient similarity search libraries for dense vectors), BatchEval (efficient mini-batch evaluation libraries), and DataManager (unified dataset management libraries). Especially, to provide a unified benchmark for the fair comparison of different complex GNN-based recommendation models, we design a new metric GRMF-X and integrate it into the FastMetrics component. Based on a TensorFlow GNN library tf_geometric, GRecX carefully implements a variety of popular GNN-based recommendation models. We carefully implement these baseline models to reproduce the performance reported in the literature, and our implementations are usually more efficient and friendly. In conclusion, GRecX enables uses to train and benchmark GNN-based recommendation baselines in an efficient and unified way. We conduct experiments with GRecX, and the experimental results show that GRecX allows us to train and benchmark GNN-based recommendation baselines in an efficient and unified way. The source code of GRecX is available at https://github.com/maenzhier/GRecX.

Motivation & Objective

• To address the lack of efficient and unified benchmarks for GNN-based recommendation models due to inconsistent evaluation practices and high computational overhead.
• To enable fair comparison across GNN-based recommendation models by standardizing evaluation protocols, especially regarding negative sampling strategies and loss functions.
• To develop high-performance, reproducible implementations of popular GNN-based recommendation models that match or exceed reported performance with improved efficiency.
• To introduce GRMF-X, a new evaluation metric that enables more effective verification of model superiority in recommendation tasks.
• To provide a unified, open-source framework that streamlines training, evaluation, and benchmarking of GNN-based recommendation models.

Proposed method

• Design and implement core libraries: FastMetrics for efficient metric computation, VectorSearch for fast similarity search on dense vectors, BatchEval for mini-batch evaluation, and DataManager for unified dataset handling.
• Integrate GRMF-X, a new evaluation metric, into FastMetrics to standardize and improve fairness in model comparison across different hyperparameter settings.
• Implement popular GNN-based recommendation models (e.g., NGCF, LightGCN, UltraGCN base) using TensorFlow, ensuring reproducibility and performance improvements over existing official implementations.
• Optimize model inference by replacing computationally expensive operations—such as full matrix multiplication for top-K retrieval—with efficient alternatives, reducing time and space complexity.
• Standardize hyperparameter settings across models, particularly negative sampling strategies (1 vs. 800 samples) and embedding dimensions, to enable fair benchmarking.
• Use tf_geometric as the underlying GNN library to ensure compatibility and efficient graph neural network operations within the framework.

Experimental results

Research questions

• RQ1 How can we design an efficient and unified benchmarking framework for GNN-based recommendation models that minimizes computational overhead?
• RQ2 To what extent does the proposed GRMF-X metric improve the fairness and effectiveness of model comparison in GNN-based recommendation?
• RQ3 Can the GRecX framework achieve faster inference and better training efficiency compared to existing implementations like RecBole and original codebases?
• RQ4 How do different negative sampling strategies and loss functions affect model performance when standardized across models in a unified benchmark?
• RQ5 To what extent can the GRecX-optimized implementations reproduce or surpass the reported performance of baseline models in the literature?

Key findings

• GRecX significantly improves inference efficiency: LightGCN implemented in GRecX achieves faster inference times than both RecBole and original code, especially on Yelp2018 and Gowalla datasets.
• The GRMF-X metric enables more effective verification of model superiority, revealing that MF and MLP+MF models outperform NGCF and LightGCN in certain settings when evaluated under fair conditions.
• When using the same negative sampling strategy (1 negative sample), LightGCN outperforms UltraGCN base_s, suggesting that performance differences in prior work may be attributed to hyperparameter imbalance rather than model architecture.
• The GRecX-optimized implementations of NGCF, LightGCN, and UltraGCN base achieve performance on par with or better than the original reports, while being more efficient and easier to reproduce.
• Standardizing hyperparameters—especially negative sampling and embedding dimensionality—reveals that MF and MLP+MF models can outperform complex GNN models in some cases, challenging assumptions about GNN superiority.
• The framework enables consistent and reproducible benchmarking across diverse models, with experimental results available at https://github.com/maenzhier/GRecX.