[论文解读] Community Detection via Random and Adaptive Sampling

Se-Young Yun, Alexandre Proutière|arXiv (Cornell University)|Feb 13, 2014

Complex Network Analysis Techniques参考文献 16被引用 29

一句话总结

本文提出了一种用于网络中社区检测的联合自适应采样与聚类框架，其中节点对之间的交互被自适应地采样，以最大化社区恢复的准确性。该研究建立了基本性能极限，并表明与非自适应策略相比，自适应采样显著降低了所需的观测预算，在预算随网络规模适当增长时，可实现渐近准确的检测。

ABSTRACT

In this paper, we consider networks consisting of a finite number of non-overlapping communities. To extract these communities, the interaction between pairs of nodes may be sampled from a large available data set, which allows a given node pair to be sampled several times. When a node pair is sampled, the observed outcome is a binary random variable, equal to 1 if nodes interact and to 0 otherwise. The outcome is more likely to be positive if nodes belong to the same communities. For a given budget of node pair samples or observations, we wish to jointly design a sampling strategy (the sequence of sampled node pairs) and a clustering algorithm that recover the hidden communities with the highest possible accuracy. We consider both non-adaptive and adaptive sampling strategies, and for both classes of strategies, we derive fundamental performance limits satisfied by any sampling and clustering algorithm. In particular, we provide necessary conditions for the existence of algorithms recovering the communities accurately as the network size grows large. We also devise simple algorithms that accurately reconstruct the communities when this is at all possible, hence proving that the proposed necessary conditions for accurate community detection are also sufficient. The classical problem of community detection in the stochastic block model can be seen as a particular instance of the problems consider here. But our framework covers more general scenarios where the sequence of sampled node pairs can be designed in an adaptive manner. The paper provides new results for the stochastic block model, and extends the analysis to the case of adaptive sampling.

研究动机与目标

• 在固定观测预算下，联合优化采样策略与聚类算法，以实现准确的社区检测。
• 分析非自适应随机采样与自适应采样策略在恢复隐藏社区方面的基本性能极限。
• 量化自适应采样相较于非自适应采样在所需观测预算方面的增益。
• 开发简单、低复杂度的算法，以实现所推导的性能极限。

提出的方法

• 提出一种框架，从大规模数据集中采样节点对之间的交互，结果表示为交互（1）或非交互（0）。
• 将交互概率建模为同社区对的p和不同社区对的q < p，允许稠密和稀疏网络情形。
• 使用测度变换论证推导误分类误差的理论下界，类似于多臂赌博机遗憾分析。
• 提出一种用于非自适应采样的谱聚类（SP）算法，构建观测矩阵并使用谱聚类。
• 设计一种自适应采样策略，根据先前结果选择节点对以最大化信息增益。
• 采用集中不等式与指数尾部界（如马尔可夫、切比雪夫及切尔诺夫型界）分析聚类误差概率。

实验结果

研究问题

• RQ1 在非自适应随机采样策略下，社区检测准确性的基本极限是什么？
• RQ2 与非自适应策略相比，自适应采样如何提升性能极限？
• RQ3 在观测预算T、网络规模n以及交互概率p和q满足何种条件时，可实现渐近准确的社区检测？
• RQ4 在非自适应与自适应设置下，能否通过简单、低复杂度的算法实现所推导的基本极限？
• RQ5 与非自适应采样相比，使用自适应采样在观测预算减少方面具有多大定量增益？

主要发现

• 对于非自适应采样，渐近准确的社区检测仅在T/n → ∞ 且 (T/n) · min{KL(q,p), KL(p,q)} → ∞ 时可能实现。
• 对于自适应采样，渐近准确检测要求 min{1−q, p} · (T/n) = Ω(1) 且 (T/n) · max{KL(q,p), KL(p,q)} → ∞。
• 所提出的谱聚类（SP）算法实现了非自适应采样下的基本下界，证明了这些界限是紧致的。
• 自适应采样显著降低了所需的观测预算，尤其在q ≪ p时效果更为明显。
• 本文证明了所推导的准确检测必要条件也是充分的，因为构造了匹配的算法。
• 分析覆盖了稠密（p,q = Θ(1)）和稀疏（p,q = o(1)）交互情形，超越了经典随机块模型的范围。