[Paper Review] Clique Counting in MapReduce: Algorithms and Experiments

Irene Finocchi, Marco Finocchi+1 more|arXiv (Cornell University)|Oct 13, 2015

Complex Network Analysis Techniques38 references29 citations

TL;DR

This paper presents the first exact, scalable MapReduce algorithm for counting k-cliques in large graphs, achieving optimal work complexity and independent communication cost from k. It also introduces highly accurate sampling-based estimators that drastically reduce runtime and space, outperforming exact methods on graphs with rapidly growing clique counts.

ABSTRACT

We tackle the problem of counting the number qk of k-cliques in large-scale graphs, for any constant k ≥ 3. Clique counting is essential in a variety of applications, including social network analysis. Our algorithms make it possible to compute qk for several real-world graphs and shed light on its growth rate as a function of k. Even for small values of k, the number qk of k-cliques can be in the order of tens or hundreds of trillions. As k increases, different graph instances show different behaviors: while on some graphs qk+1 < qk, on other benchmarks qk+1 qk, up to two orders of magnitude in our observations. Graphs with steep clique growth rates represent particularly tough instances in practice. Due to the computationally intensive nature of the clique counting problem, we settle for parallel solutions in the MapReduce framework, which has become in the last few years a de facto standard for batch processing of massive datasets. We give both theoretical and experimental contributions. On the theory side, we design the first exact scalable algorithm for counting (and listing) k-cliques in MapReduce. Our algorithm uses O(m3/2) total space and O(mk/2) work, where m is the number of graph edges. This matches the best-known bounds for triangle listing when k = 3 and is work optimal in the worst case for any k, while keeping the communication cost independent of k. We also design sampling-based estimators that can dramatically reduce the running time and space requirements of the exact approach, while providing very accurate solutions with high probability. We then assess the effectiveness of different clique counting approaches through an extensive experimental analysis over the Amazon EC2 platform, considering both our algorithms and their state-of-the-art competitors. The experimental results clearly highlight the algorithm of choice in different scenarios and prove our exact approach to be the most effective when the number of k-cliques is large, gracefully scaling to nontrivial values of k even on clusters of small/medium size. Our approximation algorithms achieve extremely accurate estimates and large speedups, especially on the toughest instances for the exact algorithms

Motivation & Objective

• Address the challenge of counting k-cliques in large-scale graphs, a critical task in social network analysis and graph mining.
• Design a scalable, exact algorithm for k-clique counting in the MapReduce framework, ensuring optimal work and communication complexity.
• Develop sampling-based estimators that provide high-accuracy approximations with significantly reduced time and space requirements.
• Evaluate the performance of exact and approximate methods across diverse real-world graphs to identify optimal strategies for different graph characteristics.
• Understand the growth behavior of k-clique counts as k increases, especially in graphs with steep clique growth rates.

Proposed method

• Design a MapReduce-based algorithm that uses O(m^3/2) total space and O(m^k/2) work, matching the best-known bounds for triangle listing (k=3) and achieving work optimality for any k.
• Structure the algorithm to ensure communication cost is independent of k, enabling scalability even for larger values of k.
• Introduce sampling-based estimators that reduce computational overhead while maintaining high accuracy with high probability.
• Leverage the MapReduce abstraction to distribute computation across clusters, enabling processing of graphs with tens to hundreds of trillions of k-cliques.
• Implement and evaluate both exact and approximate algorithms on Amazon EC2, comparing them with state-of-the-art competitors under varying graph workloads.

Experimental results

Research questions

• RQ1 How can k-clique counting be made both scalable and exact in the MapReduce model, with optimal work and communication complexity?
• RQ2 What is the behavior of k-clique growth rates across different real-world graphs, and how does it affect algorithmic performance?
• RQ3 To what extent can sampling-based estimators reduce runtime and space usage while maintaining high accuracy in k-clique counting?
• RQ4 How do the proposed exact and approximate algorithms compare in practice across diverse graph workloads and cluster sizes?
• RQ5 Which algorithm is most effective for graphs with steep clique growth rates, where exact counting becomes computationally prohibitive?

Key findings

• The proposed exact MapReduce algorithm achieves optimal work complexity O(m^k/2) and maintains communication cost independent of k, making it scalable for large k.
• On graphs with rapidly growing clique counts—where qk+1 can be up to two orders of magnitude larger than qk—the exact algorithm gracefully scales even on small-to-medium clusters.
• The sampling-based estimators achieve extremely accurate estimates with large speedups, especially on the most challenging instances for exact algorithms.
• Experimental results show that the exact algorithm outperforms state-of-the-art competitors when the number of k-cliques is large, particularly for k ≥ 4.
• The study reveals that different graphs exhibit divergent clique growth behaviors, with some showing qk+1 < qk and others showing qk+1 ≫ qk, highlighting the need for adaptive algorithm selection.