[论文解读] Clique Counting in MapReduce: Algorithms and Experiments

Irene Finocchi, Marco Finocchi+1 more|arXiv (Cornell University)|Oct 13, 2015

Complex Network Analysis Techniques参考文献 38被引用 29

一句话总结

本文提出了首个在大规模图中精确计数k-团的可扩展MapReduce算法，实现了最优的工作复杂度，并且通信成本与k无关。此外，还引入了高度准确的基于采样的估计器，显著降低了运行时间和空间开销，在团数快速增长的图上优于精确方法。

ABSTRACT

We tackle the problem of counting the number qk of k-cliques in large-scale graphs, for any constant k ≥ 3. Clique counting is essential in a variety of applications, including social network analysis. Our algorithms make it possible to compute qk for several real-world graphs and shed light on its growth rate as a function of k. Even for small values of k, the number qk of k-cliques can be in the order of tens or hundreds of trillions. As k increases, different graph instances show different behaviors: while on some graphs qk+1 < qk, on other benchmarks qk+1 qk, up to two orders of magnitude in our observations. Graphs with steep clique growth rates represent particularly tough instances in practice. Due to the computationally intensive nature of the clique counting problem, we settle for parallel solutions in the MapReduce framework, which has become in the last few years a de facto standard for batch processing of massive datasets. We give both theoretical and experimental contributions. On the theory side, we design the first exact scalable algorithm for counting (and listing) k-cliques in MapReduce. Our algorithm uses O(m3/2) total space and O(mk/2) work, where m is the number of graph edges. This matches the best-known bounds for triangle listing when k = 3 and is work optimal in the worst case for any k, while keeping the communication cost independent of k. We also design sampling-based estimators that can dramatically reduce the running time and space requirements of the exact approach, while providing very accurate solutions with high probability. We then assess the effectiveness of different clique counting approaches through an extensive experimental analysis over the Amazon EC2 platform, considering both our algorithms and their state-of-the-art competitors. The experimental results clearly highlight the algorithm of choice in different scenarios and prove our exact approach to be the most effective when the number of k-cliques is large, gracefully scaling to nontrivial values of k even on clusters of small/medium size. Our approximation algorithms achieve extremely accurate estimates and large speedups, especially on the toughest instances for the exact algorithms

研究动机与目标

• 解决大规模图中k-团计数的挑战，这是社交网络分析和图挖掘中的关键任务。
• 在MapReduce框架中设计一种可扩展的精确算法，实现k-团计数，确保最优的工作复杂度和通信复杂度。
• 开发基于采样的估计器，提供高精度近似结果，同时显著降低时间和空间需求。
• 在多种真实世界图上评估精确与近似方法的性能，以识别不同图特征下的最优策略。
• 理解k-团计数随k增加的增长行为，特别是在团数快速增长的图中。

提出的方法

• 设计一种基于MapReduce的算法，使用O(m^3/2)总空间和O(m^k/2)工作量，与三角形列出（k=3）的最佳已知界限一致，并对任意k实现工作量最优。
• 通过算法结构设计，确保通信成本与k无关，从而在k较大时仍保持可扩展性。
• 引入基于采样的估计器，在保持高概率高精度的同时降低计算开销。
• 利用MapReduce抽象将计算分布到集群中，支持处理包含数十亿至数万亿个k-团的图。
• 在Amazon EC2上实现并评估精确与近似算法，与最先进的竞争算法在不同图工作负载下进行比较。

实验结果

研究问题

• RQ1 如何在MapReduce模型中实现k-团计数的可扩展性与精确性，同时达到最优的工作复杂度和通信复杂度？
• RQ2 不同真实世界图中k-团的增长率行为如何？其对算法性能有何影响？
• RQ3 基于采样的估计器在k-团计数中能在多大程度上减少运行时间和空间使用，同时保持高精度？
• RQ4 在不同图工作负载和集群规模下，所提出的精确与近似算法在实际应用中的表现如何比较？
• RQ5 对于团数快速增长的图（精确计数在计算上变得不可行），哪种算法最为有效？

主要发现

• 所提出的精确MapReduce算法实现了最优工作复杂度O(m^k/2)，并保持通信成本与k无关，使其在大k值下仍具可扩展性。
• 在团数快速增长的图上（其中qk+1可比qk高出两个数量级），该精确算法即使在小型到中型集群上也能平稳扩展。
• 基于采样的估计器实现了极高的估计精度，并带来显著的速度提升，尤其在精确算法处理最困难实例时表现突出。
• 实验结果表明，当k-团数量较大时，精确算法优于最先进的竞争算法，尤其在k ≥ 4时表现更优。
• 研究发现，不同图表现出截然不同的团增长行为，部分图呈现qk+1 < qk，而另一些图则呈现qk+1 ≫ qk，凸显了自适应算法选择的必要性。