[Paper Review] FlashGraph: processing billion-node graphs on an array of commodity SSDs

Da Zheng, Disa Mhembere+4 more|arXiv (Cornell University)|Feb 16, 2015

Graph Theory and Algorithms31 references185 citations

TL;DR

FlashGraph enables efficient processing of billion-node graphs on commodity SSDs by storing vertex states in memory and edge lists on SSDs, using a user-space SSD file system to achieve high IOPS and parallel I/O. It matches up to 80% of in-memory performance and outperforms PowerGraph by minimizing I/O overhead through request merging and latency hiding.

ABSTRACT

Graph analysis performs many random reads and writes, thus, these workloads are typically performed in memory. Traditionally, analyzing large graphs requires a cluster of machines so the aggregate memory exceeds the graph size. We demonstrate that a multicore server can process graphs with billions of vertices and hundreds of billions of edges, utilizing commodity SSDs with minimal performance loss. We do so by implementing a graph-processing engine on top of a user-space SSD file system designed for high IOPS and extreme parallelism. Our semi-external memory graph engine called FlashGraph stores vertex state in memory and edge lists on SSDs. It hides latency by overlapping computation with I/O. To save I/O bandwidth, FlashGraph only accesses edge lists requested by applications from SSDs; to increase I/O throughput and reduce CPU overhead for I/O, it conservatively merges I/O requests. These designs maximize performance for applications with different I/O characteristics. FlashGraph exposes a general and flexible vertex-centric programming interface that can express a wide variety of graph algorithms and their optimizations. We demonstrate that FlashGraph in semi-external memory performs many algorithms with performance up to 80% of its in-memory implementation and significantly outperforms PowerGraph, a popular distributed in-memory graph engine.

Motivation & Objective

• Address the scalability bottleneck of in-memory graph processing for billion-node graphs.
• Enable large-scale graph analysis on a single multicore server instead of a cluster.
• Reduce reliance on expensive main memory by leveraging commodity SSDs effectively.
• Minimize I/O overhead and CPU cost while maintaining high throughput for diverse graph workloads.
• Design a flexible, vertex-centric programming interface for expressing various graph algorithms.

Proposed method

• Implement a user-space SSD file system optimized for high IOPS and extreme parallelism to support low-latency I/O operations.
• Store vertex states in main memory and edge lists on SSDs, enabling semi-external memory processing.
• Overlap computation with I/O to hide latency, improving overall throughput.
• Limit SSD access to only requested edge lists, reducing I/O bandwidth usage.
• Conservatively merge I/O requests to increase throughput and reduce CPU overhead.
• Expose a general vertex-centric programming interface to express diverse graph algorithms and optimizations.

Experimental results

Research questions

• RQ1 Can a single-server system with commodity SSDs process billion-node graphs efficiently without requiring in-memory storage?
• RQ2 How effectively can I/O latency be hidden in graph processing using overlapping computation and I/O?
• RQ3 To what extent can I/O bandwidth be reduced by accessing only requested edge lists?
• RQ4 How does conservative I/O request merging impact throughput and CPU overhead in graph workloads?
• RQ5 How does FlashGraph's performance compare to in-memory systems and existing distributed engines like PowerGraph?

Key findings

• FlashGraph achieves up to 80% of the performance of its in-memory counterpart on various graph algorithms.
• FlashGraph significantly outperforms PowerGraph, a leading distributed in-memory graph engine, on the same workloads.
• The system effectively hides I/O latency by overlapping computation with I/O operations.
• By accessing only requested edge lists, FlashGraph reduces unnecessary I/O bandwidth consumption.
• Conservative merging of I/O requests increases I/O throughput while minimizing CPU overhead.
• The semi-external memory design enables processing of graphs with billions of vertices and hundreds of billions of edges on a single server.