[Paper Review] Community Detection by Information Flow Simulation

Rajagopal Venkatesaramani, Yevgeniy Vorobeychik|arXiv (Cornell University)|May 13, 2018

Complex Network Analysis Techniques17 references3 citations

TL;DR

This paper proposes a fast, scalable community detection algorithm that simulates information flow in directed, weighted graphs by modeling edges as probabilistic paths. It achieves $Ó(|E|)$ time and space complexity, outperforming MCL in both speed and accuracy on large-scale networks like YouTube and Wikipedia, where MCL fails to scale.

ABSTRACT

Community detection remains an important problem in data mining, owing to the lack of scalable algorithms that exploit all aspects of available data - namely the directionality of flow of information and the dynamics thereof. Most existing methods use measures of connectedness in the graphical structure. In this paper, we present a fast, scalable algorithm to detect communities in directed, weighted graph representations of social networks by simulating flow of information through them. By design, our algorithm naturally handles undirected or unweighted networks as well. Our algorithm runs in $\mathcal{O}(|E|)$ time, which is better than most existing work and uses $\mathcal{O}(|E|)$ space and hence scales easily to very large datasets. Finally, we show that our algorithm outperforms the state-of-the-art Markov Clustering Algorithm (MCL) in both accuracy and scalability on ground truth data (in a number of cases, we can find communities in graphs too large for MCL).

Motivation & Objective

• Address the lack of scalable community detection algorithms that exploit directionality and dynamics of information flow in social networks.
• Overcome the limitations of existing methods that rely on static, connectivity-based measures and fail to scale to large, real-world networks.
• Develop a dynamic, flow-based approach that naturally handles undirected and unweighted networks while maintaining high performance.
• Achieve linear time and space complexity to enable processing of very large datasets, such as those from Facebook, YouTube, and Wikipedia.
• Demonstrate superior performance over state-of-the-art methods like MCL on both accuracy and scalability, especially on challenging datasets with complex community structures.

Proposed method

• Represent social networks as directed, weighted graphs where edge weights reflect the probability of information flow between nodes.
• Simulate information flow from each node by treating edges as probabilistic transfer paths, using a dynamic data structure instead of sparse matrices.
• Use a randomized, iterative process to simulate flow propagation, identifying influential nodes that act as information origins.
• Apply a clustering mechanism based on flow convergence, grouping nodes that receive significant information flow from common sources.
• Leverage the small-world effect and community structure properties to bound the expected running time and ensure efficiency.
• Optimize for large-scale graphs by ensuring linear time complexity ($Ó(|E|)$) and linear space usage ($Ó(|E|)$), enabling processing of graphs with millions of edges.

Experimental results

Research questions

• RQ1 Can information flow simulation in directed, weighted graphs effectively detect communities while maintaining linear time and space complexity?
• RQ2 How does the proposed algorithm compare to MCL in terms of accuracy and scalability on real-world, large-scale networks?
• RQ3 To what extent does the algorithm perform on datasets where prior methods, including MCL, fail to detect meaningful community structures?
• RQ4 How do the algorithm’s performance metrics (e.g., false positive/negative rates, conductance) vary across different network types and cluster sizes?
• RQ5 Can the algorithm be effectively tuned via the parameter $k$ to detect communities of varying sizes and densities, especially when ground truth is unavailable?

Key findings

• The algorithm runs in $Ó(|E|)$ time and space, making it highly scalable and capable of processing graphs with up to 28.5 million edges, such as the Wikipedia dataset.
• On the YouTube dataset—previously challenging for community detection—our algorithm achieves a false positive rate of 0.15 and a false negative rate below 0.02 with unbiased sampling.
• For the DBLP and Email Eu Core networks, our method outperforms MCL in accuracy, despite MCL achieving lower conductance scores, indicating conductance alone is not a reliable metric.
• On the Wikipedia network (28.5M edges), our algorithm completes in 3 hours and 4 minutes, while MCL fails to scale and does not complete.
• On the YouTube network (5.98M edges), our algorithm runs in 1 hour and 7 minutes, whereas MCL does not scale and cannot process the dataset.
• The algorithm’s runtime scales almost linearly with the number of edges, confirming the theoretical upper bound derived in Theorem 5.4.