[Paper Review] Community Detection in Sparse Random Networks

Ery Arias-Castro, Nicolas Verzélen|arXiv (Cornell University)|Aug 13, 2013

Complex Network Analysis Techniques4 citations

TL;DR

This paper studies community detection in sparse random networks by testing for the presence of a hidden dense subgraph (planted community) within an Erdős-Rényi random graph. It establishes sharp detection thresholds using information-theoretic lower bounds and analyzes multiple test statistics—including the scan statistic, largest connected component, number of triangles, and subtree counts—providing near-optimal detection performance in the sparse regime where edge probabilities decay with network size.

ABSTRACT

We consider the problem of detecting a tight community in a sparse random network. This is formalized as testing for the existence of a dense random subgraph in a random graph. Under the null hypothesis, the graph is a realization of an Erdös-Rényi graph on $N$ vertices and with connection probability $p_0$; under the alternative, there is an unknown subgraph on $n$ vertices where the connection probability is p1 > p0. In Arias-Castro and Verzelen (2012), we focused on the asymptotically dense regime where p0 is large enough that np0>(n/N)^{o(1)}. We consider here the asymptotically sparse regime where p0 is small enough that np00. As before, we derive information theoretic lower bounds, and also establish the performance of various tests. Compared to our previous work, the arguments for the lower bounds are based on the same technology, but are substantially more technical in the details; also, the methods we study are different: besides a variant of the scan statistic, we study other statistics such as the size of the largest connected component, the number of triangles, the eigengap of the adjacency matrix, etc. Our detection bounds are sharp, except in the Poisson regime where we were not able to fully characterize the constant arising in the bound.

Motivation & Objective

• To address the problem of detecting a hidden, dense subgraph (community) in a sparse random network where edge probabilities decay with size.
• To extend prior work on dense regimes to the asymptotically sparse regime, where $ np_0 < (n/N)^{c_0} $ for some $ c_0 > 0 $.
• To derive information-theoretic lower bounds on detection performance under the sparse regime.
• To evaluate and compare the power of multiple test statistics, including scan, component size, triangles, and subtrees, for community detection.
• To establish sharp detection thresholds, except in the Poisson regime where the constant in the bound remains uncharacterized.

Proposed method

• Formalizes the problem as minimax hypothesis testing: $ H_0 $ (Erdős-Rényi graph with edge probability $ p_0 $) vs. $ H_1 $ (a hidden subgraph of size $ n $ with higher edge probability $ p_1 > p_0 $).
• Uses the scan statistic as a primary test, evaluating its performance via concentration and tail bounds on the number of edges in subgraphs.
• Analyzes alternative statistics: size of the largest connected component, number of triangles, and number of subtrees of a given size.
• Applies Stirling's approximation and combinatorial bounds to control the number of configurations and edge probabilities in subgraphs.
• Derives lower bounds using a change-of-measure technique and the second moment method, focusing on rare but informative subgraph configurations.
• Employs asymptotic analysis under the regime $ N \to \infty $, $ n \to \infty $, $ n/N \to 0 $, $ n/\log N \to \infty $, with $ p_0, p_1 \to 0 $.

Experimental results

Research questions

• RQ1 What is the fundamental detection limit for identifying a hidden dense subgraph in a sparse random network?
• RQ2 How do different test statistics—scan, largest component, triangles, subtree counts—compare in performance under the sparse regime?
• RQ3 Can sharp detection thresholds be established in the sparse regime, particularly when $ p_0 $ and $ p_1 $ decay with $ N $?
• RQ4 Why does the Poisson regime remain unresolved, and what prevents full characterization of the detection threshold there?
• RQ5 How do the theoretical detection limits compare to practical test statistics in terms of achievable power?

Key findings

• The paper establishes sharp detection thresholds for community detection in the sparse regime, with the scan statistic achieving near-optimal performance.
• The detection boundary is characterized by a threshold involving $ p_1 $, $ p_0 $, $ n $, and $ N $, with the bound being tight except in the Poisson regime.
• The number of triangles and the size of the largest connected component are shown to be effective detection statistics, though less powerful than the scan statistic in general.
• The subtree-counting statistic is analyzed via combinatorial enumeration and moment bounds, showing its utility in sparse settings.
• The lower bounds are derived using a second-moment method on carefully chosen subgraph configurations, demonstrating the information-theoretic limit of detection.
• In the Poisson regime, the constant in the detection threshold remains uncharacterized, indicating a gap in the theoretical understanding of this case.