[Paper Review] In-Depth Benchmarking of Graph Database Systems with the Linked Data Benchmark Council (LDBC) Social Network Benchmark (SNB)

Florin Rusu, Zhiyi Huang|arXiv (Cornell University)|Jul 17, 2019

Graph Theory and Algorithms12 references4 citations

TL;DR

This paper presents a comprehensive, reproducible benchmark of Neo4j and TigerGraph using the LDBC Social Network Benchmark (SNB), evaluating performance across 46 queries at scale factors SF-1 to SF-1000 on on-premise and cloud architectures. TigerGraph consistently outperforms Neo4j by 2+ orders of magnitude on most queries, especially complex and business intelligence workloads, with a 4X storage reduction and superior scalability to SF-1000, while Neo4j is faster at raw data loading up to SF-100 despite slower index building.

ABSTRACT

In this study, we present the first results of a complete implementation of the LDBC SNB benchmark -- interactive short, interactive complex, and business intelligence -- in two native graph database systems---Neo4j and TigerGraph. In addition to thoroughly evaluating the performance of all of the 46 queries in the benchmark on four scale factors -- SF-1, SF-10, SF-100, and SF-1000 -- and three computing architectures -- on premise and in the cloud -- we also measure the bulk loading time and storage size. Our results show that TigerGraph is consistently outperforming Neo4j on the majority of the queries---by two or more orders of magnitude (100X factor) on certain interactive complex and business intelligence queries. The gap increases with the size of the data since only TigerGraph is able to scale to SF-1000---Neo4j finishes only 12 of the 25 business intelligence queries in reasonable time. Nonetheless, Neo4j is generally faster at bulk loading graph data up to SF-100. A key to our study is the active involvement of the vendors in the tuning of their platforms. In order to encourage reproducibility, we make all the code, scripts, and configuration parameters publicly available online.

Motivation & Objective

• To provide a complete, reproducible evaluation of native graph database systems using the LDBC SNB benchmark.
• To compare the performance of Neo4j and TigerGraph across interactive, complex, and business intelligence workloads at scale factors SF-1 to SF-1000.
• To measure and contrast bulk loading times, storage efficiency, and query execution performance on on-premise and cloud deployments.
• To involve system vendors in query tuning and ensure reproducibility through public release of all code, scripts, and configurations.
• To establish a reference implementation of all 46 LDBC SNB queries in Cypher and GSQL for future benchmarking.

Proposed method

• Full implementation of all 46 LDBC SNB queries in both Neo4j (Cypher) and TigerGraph (GSQL), with vendor-guided optimization.
• Execution of all queries across four scale factors (SF-1, SF-10, SF-100, SF-1000) on three computing architectures: on-premise and two cloud environments.
• Measurement of query execution time, bulk loading time, and storage size (raw and indexed) for both systems.
• Use of the official LDBC SNB data generator to produce synthetic social network data with realistic schema and distribution.
• Application of timeout thresholds (18,000 seconds) to identify non-scalable query performance.
• Public release of all query scripts, configuration files, and benchmarking code to ensure reproducibility.

Experimental results

Research questions

• RQ1 How do Neo4j and TigerGraph perform across the full spectrum of LDBC SNB workloads—interactive short, interactive complex, and business intelligence—across multiple scale factors?
• RQ2 What is the relative performance of Neo4j and TigerGraph in terms of query execution time, bulk loading, and storage efficiency?
• RQ3 To what extent can each system scale to large-scale data (SF-1000), and what are the performance bottlenecks?
• RQ4 How does the inclusion of vendor-optimized query plans affect the outcome and reproducibility of the benchmark?
• RQ5 What are the trade-offs between query performance, data loading speed, and storage footprint in native graph databases?

Key findings

• TigerGraph outperforms Neo4j on 96.5% of the 368 benchmark configurations, with more than 95% of workloads showing superior performance.
• TigerGraph achieves more than 100X faster execution on certain interactive complex and business intelligence queries, with performance gaps increasing at larger scale factors.
• Only TigerGraph successfully scales to SF-1000; Neo4j fails to complete 13 out of 25 business intelligence queries within the 18,000-second timeout.
• TigerGraph uses 3X less raw storage and 4X less indexed storage than Neo4j, and compresses the original data by a factor of 2X.
• Neo4j is faster at bulk loading raw data (up to 3X faster at SF-1) but suffers from non-scalable index building, making total load time 2X slower at SF-1000.
• Despite its more compact syntax, Neo4j’s query execution is outperformed by TigerGraph in nearly all workloads, indicating architectural advantages in query processing.