[Paper Review] A Distributed Force-Directed Algorithm on Giraph: Design and Experiments

Alessio Arleo, Walter Didimo+2 more|arXiv (Cornell University)|Jun 7, 2016

Data Visualization and Analytics33 references3 citations

TL;DR

This paper presents GiLA, a distributed force-directed graph layout algorithm implemented in Apache Giraph using a vertex-centric 'Think-Like-A-Vertex' (TLAV) paradigm. It achieves scalable, low-cost visualization of large graphs—up to one million edges—in under 8 minutes at a cost of approximately $1 on Amazon's PaaS, outperforming centralized implementations while enabling effective visual cluster detection via the LinLog model and K-means clustering.

ABSTRACT

In this paper we study the problem of designing a distributed graph visualization algorithm for large graphs. The algorithm must be simple to implement and the computing infrastructure must not require major hardware or software investments. We design, implement, and experiment a force-directed algorithm in Giraph, a popular open source framework for distributed computing, based on a vertex-centric design paradigm. The algorithm is tested both on real and artificial graphs with up to million edges, by using a rather inexpensive PaaS (Platform as a Service) infrastructure of Amazon. The experiments show the scalability and effectiveness of our technique when compared to a centralized implementation of the same force-directed model. We show that graphs with about one million edges can be drawn in less than 8 minutes, by spending about 1\$ per drawing in the cloud computing infrastructure.

Motivation & Objective

• To design a scalable, low-cost distributed graph visualization algorithm suitable for large-scale graphs.
• To implement a force-directed layout algorithm using the TLAV paradigm in Apache Giraph for efficient distributed execution.
• To evaluate the algorithm's performance and cost-effectiveness on real and synthetic graphs using an inexpensive cloud infrastructure.
• To enable visual cluster detection in large graphs by adapting the LinLog force model and applying K-means clustering to node positions.
• To provide open-source code for reuse in future research on distributed graph visualization.

Proposed method

• The algorithm is based on the Fruchterman-Reingold force-directed model, adapted for distributed execution using the TLAV paradigm in Giraph.
• Each vertex independently computes forces from its neighbors and updates its position iteratively in a vertex-centric manner.
• The force model is extended to support the LinLog energy function for enhanced cluster visualization.
• A K-means clustering algorithm is applied to the final node coordinates to detect and color clusters.
• The implementation leverages Apache Giraph’s fault-tolerant, scalable execution model on a PaaS cloud infrastructure.
• The algorithm is evaluated on real-world and synthetic graphs with up to one million edges using a 21-node Amazon EMR cluster.

Experimental results

Research questions

• RQ1 Can a force-directed graph layout algorithm be effectively and efficiently distributed using a vertex-centric model on a cloud-based PaaS infrastructure?
• RQ2 How does the performance and cost of the distributed algorithm compare to a centralized implementation for large graphs?
• RQ3 To what extent does the LinLog force model improve visual cluster detection in large graphs when combined with distributed layout and K-means clustering?
• RQ4 What is the scalability and runtime cost of the algorithm on graphs with up to one million edges in a cloud environment?
• RQ5 Can the TLAV-based design achieve linear scalability and maintain layout quality across diverse graph topologies?

Key findings

• Graphs with up to one million edges were successfully visualized in less than 8 minutes using the GiLA algorithm on a 21-node Amazon EMR cluster.
• The total cost of rendering such graphs was approximately $1 USD per drawing, demonstrating low-cost operation on a PaaS infrastructure.
• The algorithm showed strong scalability and performance improvements over centralized implementations, particularly on graphs with small diameters.
• The LinLog force model, combined with K-means clustering, produced high-quality visual clusters, with modularity averaging 0.51 and conductance averaging 0.57 across tested graphs.
• The Calinski-Harabasz index was used to automatically select optimal cluster counts, and the resulting clustering quality was consistently high across multiple metrics.
• The implementation was made publicly available at http://www.geeksykings.eu/gila/, enabling reuse and extension in future research.