[Paper Review] PMV: Pre-partitioned Generalized Matrix-Vector Multiplication for Scalable Graph Mining

Chiwan Park, Ha-Myung Park+2 more|arXiv (Cornell University)|Sep 26, 2017

Graph Theory and Algorithms41 references3 citations

TL;DR

PMV proposes a pre-partitioned generalized matrix-vector multiplication framework for scalable graph mining on distributed systems. By partitioning the input graph in advance and applying adaptive horizontal/vertical block placement based on submatrix density, PMV reduces communication and I/O costs, enabling processing of graphs up to 16× larger than previous distributed-memory methods and achieving 9× faster performance than disk-based systems.

ABSTRACT

How can we analyze enormous networks including the Web and social networks which have hundreds of billions of nodes and edges? Network analyses have been conducted by various graph mining methods including shortest path computation, PageRank, connected component computation, random walk with restart, etc. These graph mining methods can be expressed as generalized matrix-vector multiplication which consists of few operations inspired by typical matrix-vector multiplication. Recently, several graph processing systems based on matrix-vector multiplication or their own primitives have been proposed to deal with large graphs; however, they all have failed on Web-scale graphs due to insufficient memory space or the lack of consideration for I/O costs. In this paper, we propose PMV (Pre-partitioned generalized Matrix-Vector multiplication), a scalable distributed graph mining method based on generalized matrix-vector multiplication on distributed systems. PMV significantly decreases the communication cost, which is the main bottleneck of distributed systems, by partitioning the input graph in advance and judiciously applying execution strategies based on the density of the pre-partitioned sub-matrices. Experiments show that PMV succeeds in processing up to 16x larger graphs than existing distributed memory-based graph mining methods, and requires 9x less time than previous disk-based graph mining methods by reducing I/O costs significantly.

Motivation & Objective

• Address the scalability limitations of existing distributed graph mining systems on Web-scale graphs with hundreds of billions of edges.
• Overcome I/O and communication bottlenecks in distributed matrix-vector multiplication by minimizing shuffling and disk access.
• Design a general-purpose framework compatible with major distributed computing platforms like Hadoop and Spark.
• Achieve linear machine scalability and handle graphs beyond the memory capacity of single machines or distributed memory systems.

Proposed method

• Pre-partition the input graph’s adjacency matrix into blocks before iterative computation to eliminate repeated partitioning.
• Classify matrix blocks into sparse and dense regions based on vertex out-degree using a threshold θ.
• Apply horizontal placement for sparse blocks and vertical placement for dense blocks to minimize I/O and communication costs.
• Use a hybrid strategy (PMVhybrid) that selects the optimal placement per block based on density, reducing I/O by up to 44% compared to vertical-only.
• Implement the framework on Hadoop and Spark, leveraging in-place updates in Hadoop and iterative optimization in Spark.
• Optimize block assignment to workers to balance load and reduce I/O, especially for high-degree vertices that cause bottlenecks in MapReduce systems.

Experimental results

Research questions

• RQ1 Can pre-partitioning the matrix reduce communication and I/O overhead in distributed generalized matrix-vector multiplication?
• RQ2 How does the choice between horizontal and vertical block placement strategies affect I/O and performance for sparse vs. dense graphs?
• RQ3 What is the optimal threshold θ for balancing I/O and computation cost in the hybrid placement strategy?
• RQ4 Can PMV achieve linear scalability with respect to the number of machines and graph size?
• RQ5 How does PMV compare to existing systems in handling Web-scale graphs like ClueWeb12?

Key findings

• PMV successfully processes the full ClueWeb12 graph (6B vertices, 71B edges), which causes out-of-memory or out-of-time failures in all competing systems.
• PMV achieves up to 16× higher scalability than existing distributed-memory systems like GraphLab and GraphX, which fail on graphs exceeding their memory capacity.
• PMV reduces I/O by 44% compared to the vertical-only strategy (PMVvertical) when θ = 200, with a corresponding 9× speedup over disk-based MapReduce systems.
• PMV shows linear machine scalability on YahooWeb, with speedup close to 1.0 per doubling of workers, outperforming PEGASUS due to avoiding the last-reducer bottleneck.
• On small graphs, PMV on Spark is faster than on Hadoop due to Spark’s iterative computation optimizations; on large graphs, PMV on Hadoop outperforms Spark due to lower memory usage from in-place updates.
• The hybrid strategy (PMVhybrid) achieves up to 18% better performance than the selective strategy (PMVselective), demonstrating the effectiveness of adaptive block placement.