[Paper Review] Scaling Fine-grained Modularity Clustering for Massive Graphs

Hiroaki Shiokawa, Toshiyuki Amagasa+1 more|arXiv (Cornell University)|May 27, 2019

Complex Network Analysis Techniques22 references4 citations

TL;DR

This paper proposes gScarf, a highly efficient algorithm for fine-grained modularity clustering in massive graphs that dynamically prunes unnecessary nodes and edges using LRM-gain caching and incremental subgraph folding. It achieves state-of-the-art speed while maintaining clustering accuracy comparable to CorMod, avoiding the resolution limit problem through likelihood-ratio modularity maximization.

ABSTRACT

Modularity clustering is an essential tool to understand complicated graphs. However, existing methods are not applicable to massive graphs due to two serious weaknesses. (1) It is difficult to fully reproduce ground-truth clusters due to the resolution limit problem. (2) They are computationally expensive because all nodes and edges must be computed iteratively. This paper proposes gScarf, which outputs fine-grained clusters within a short running time. To overcome the aforementioned weaknesses, gScarf dynamically prunes unnecessary nodes and edges, ensuring that it captures fine-grained clusters. Experiments show that gScarf outperforms existing methods in terms of running time while finding clusters with high accuracy.

Motivation & Objective

• To address the resolution limit problem in modularity clustering, which causes coarse-grained clusters in massive graphs.
• To reduce the high computational cost of existing modularity clustering methods, especially for billion-edge graphs.
• To develop an efficient algorithm that maintains the high clustering quality of CorMod while significantly improving runtime.
• To enable practical deployment of fine-grained modularity clustering on real-world massive graphs without user-specified parameters.

Proposed method

• gScarf introduces an incremental form of likelihood-ratio modularity (LRM), called LRM-gain, to enable efficient, incremental computation of cluster quality.
• It employs LRM-gain caching based on the deterministic nature of LRM, avoiding redundant computations for clusters with identical structural properties.
• The algorithm uses incremental subgraph folding to merge nodes and edges dynamically, reducing the search space during clustering.
• It applies deterministic pruning by skipping LRM-gain computations for nodes and edges that cannot improve the current clustering.
• The method is designed to be equivalent to CorMod’s greedy LRM maximization, ensuring no loss in clustering quality.
• It supports directed graphs by modifying degree calculations to account for in-degree and out-degree separately in LRM-gain and folding operations.

Experimental results

Research questions

• RQ1 Can we achieve fine-grained modularity clustering in massive graphs without sacrificing accuracy compared to state-of-the-art methods?
• RQ2 How can we reduce the computational cost of LRM maximization in massive graphs while preserving the ability to detect ground-truth cluster sizes?
• RQ3 What dynamic pruning strategies can be applied to LRM-based clustering without missing opportunities to increase modularity?
• RQ4 Can LRM-gain be computed incrementally and cached effectively to accelerate clustering in large-scale graphs?
• RQ5 How does the proposed method scale across different graph sizes and structural complexities, especially under high mixing parameters?

Key findings

• gScarf achieves higher NMI scores than Louvain and IncMod on real-world graphs (YT, LJ, OK), approaching or slightly exceeding CorMod’s performance.
• On LFR-benchmark graphs with varying mixing parameters (mu), gScarf maintains high NMI scores even at mu = 0.9, while other methods degrade significantly.
• For large graphs with up to 10^7 nodes, gScarf consistently outperforms other methods in NMI, demonstrating robustness to scale.
• gScarf successfully reproduces ground-truth cluster sizes in both real and synthetic graphs, confirming its ability to avoid the resolution limit problem.
• The algorithm runs significantly faster than CorMod and other state-of-the-art methods, with theoretical time complexity lower than O(nm log n).
• gScarf’s LRM-gain caching and incremental folding techniques ensure no loss in clustering quality, as proven by theoretical lemmas guaranteeing no missed improvement opportunities.