[Paper Review] The GAP Benchmark Suite

Scott Beamer, Krste Asanović+1 more|arXiv (Cornell University)|Aug 14, 2015

Graph Theory and AlgorithmsComputer Science160 citations

TL;DR

The GAP Benchmark Suite introduces a standardized, diverse benchmark for graph processing that specifies kernels, input graphs, evaluation methods, and optimized reference implementations to enable fair, reproducible comparisons across research in graph algorithms, frameworks, and hardware. It supports shared-memory platforms and promotes performance improvements by setting a high bar via state-of-the-art reference code.

ABSTRACT

We present a graph processing benchmark suite with the goal of helping to standardize graph processing evaluations. Fewer differences between graph processing evaluations will make it easier to compare different research efforts and quantify improvements. The benchmark not only specifies graph kernels, input graphs, and evaluation methodologies, but it also provides optimized baseline implementations. These baseline implementations are representative of state-of-the-art performance, and thus new contributions should outperform them to demonstrate an improvement. The input graphs are sized appropriately for shared memory platforms, but any implementation on any platform that conforms to the benchmark's specifications could be compared. This benchmark suite can be used in a variety of settings. Graph framework developers can demonstrate the generality of their programming model by implementing all of the benchmark's kernels and delivering competitive performance on all of the benchmark's graphs. Algorithm designers can use the input graphs and the baseline implementations to demonstrate their contribution. Platform designers and performance analysts can use the suite as a workload representative of graph processing.

Motivation & Objective

• Address the lack of standardization in graph processing evaluations, which hinders fair comparison between research contributions.
• Reduce methodological inconsistencies—such as differences in edge directionality or termination criteria—that distort performance comparisons.
• Provide a common, high-quality baseline using optimized reference implementations to discourage low-performance baselines and encourage meaningful improvements.
• Support diverse stakeholders: framework developers, algorithm designers, and hardware architects, by offering a unified evaluation framework.
• Expand beyond existing benchmarks like Graph 500 by including multiple kernels and real-world graphs, not just BFS on synthetic Kronecker graphs.

Proposed method

• Define a suite of six graph kernels: BFS, PageRank, Connected Components, Betweenness Centrality, Triangle Counting, and SSSP, each with precise semantics and correctness requirements.
• Specify a diverse set of real-world input graphs (e.g., from SNAP, NCI, and other sources) sized for shared-memory systems, ensuring representativeness and avoiding trivial or synthetic-only datasets.
• Provide optimized, production-quality reference implementations in C++/OpenMP that achieve state-of-the-art performance, serving as a performance baseline for new contributions.
• Enforce correctness through verification routines: serial implementations for output comparison, tolerance checks for PageRank, and label equivalence checks for Connected Components.
• Use deterministic execution and consistent timing practices (e.g., excluding graph loading time) to ensure reproducibility and fairness in benchmarking.
• Support extensibility by decoupling the benchmark specification from the reference implementation, enabling any compliant implementation on any platform to be compared.

Experimental results

Research questions

• RQ1 How can a standardized benchmark suite improve the reproducibility and comparability of graph processing research across different frameworks and platforms?
• RQ2 To what extent do methodological differences—such as edge directionality or termination conditions—affect performance measurements in graph kernels?
• RQ3 Can a diverse set of real-world graphs and multiple kernels expose weaknesses in optimizations that are only effective on specific topologies?
• RQ4 How effective is a high-performance reference implementation in setting a meaningful performance baseline that motivates genuine algorithmic or system-level improvements?
• RQ5 In what ways can a standardized benchmark reduce reliance on synthetic or trivially small graphs that do not reflect real workloads?

Key findings

• The benchmark suite provides a standardized, well-specified evaluation framework that enables fair comparison across graph processing systems, frameworks, and hardware platforms.
• The inclusion of optimized reference implementations sets a high performance bar, ensuring that new contributions must outperform state-of-the-art to demonstrate real improvement.
• Verification routines—such as serial output comparison and tolerance checks—ensure correctness and reduce the risk of incorrect or misleading results.
• The suite supports multiple kernels and real-world graphs, exposing limitations of optimizations that only work on specific graph topologies or synthetic structures.
• The benchmark is platform-agnostic: any implementation conforming to the specification can be evaluated, even without using the reference code.
• The reference implementation includes performance optimizations such as early termination in triangle counting and bitmap-based successor storage in Betweenness Centrality, demonstrating practical high-performance engineering.