[Paper Review] All-Distances Sketches, Revisited: Scalable Estimation of the Distance Distribution and Centralities in Massive Graphs

Edith Cohen|arXiv (Cornell University)|Jun 14, 2013

Data Management and Algorithms22 references7 citations

TL;DR

This paper introduces the Historic Inverse Probability (HIP) estimators for All-Distances Sketches (ADS), enabling highly accurate, unbiased, and low-variance estimation of distance-related graph statistics in massive graphs. HIP reduces variance by up to half compared to prior estimators and improves estimation quality in HyperLogLog MinHash sketches, offering a unified, scalable framework for centrality and neighborhood cardinality computation.

ABSTRACT

Graph datasets with billions of edges, such as social and Web graphs, are prevalent, and scalable computation is critical. All-distances sketches (ADS) [Cohen 1997], are a powerful tool for scalable approximation of statistics. The sketch is a small size sample of the distance relation of a node which emphasizes closer nodes. Sketches for all nodes are computed using a nearly linear computation and estimators are applied to sketches of nodes to estimate their properties. We provide, for the first time, a unified exposition of ADS algorithms and applications. We present the Historic Inverse Probability (HIP) estimators which are applied to the ADS of a node to estimate a large natural class of statistics. For the important special cases of neighborhood cardinalities (the number of nodes within some query distance) and closeness centralities, HIP estimators have at most half the variance of previous estimators and we show that this is essentially optimal. Moreover, HIP obtains a polynomial improvement for more general statistics and the estimators are simple, flexible, unbiased, and elegant. For approximate distinct counting on data streams, HIP outperforms the original estimators for the HyperLogLog MinHash sketches (Flajolet et al. 2007), obtaining significantly improved estimation quality for this state-of-the-art practical algorithm.

Motivation & Objective

• To provide a unified exposition of All-Distances Sketches (ADS) algorithms and their applications in massive graph analysis.
• To address the lack of low-variance, unbiased estimators for distance-related statistics in large-scale graphs.
• To improve estimation accuracy for neighborhood cardinalities and closeness centralities, which are critical for graph centrality analysis.
• To extend the applicability of ADS to a broad class of graph statistics through a general and flexible estimator framework.
• To enhance the performance of approximate distinct counting in data streams, particularly for HyperLogLog MinHash sketches.

Proposed method

• Proposes the Historic Inverse Probability (HIP) estimator, which leverages historical sampling probabilities to reduce variance in distance statistics estimation.
• Applies HIP to All-Distances Sketches (ADS), where each node’s sketch captures a biased sample favoring closer nodes, enabling efficient computation.
• Designs estimators that are unbiased and exhibit significantly lower variance than prior methods, particularly for neighborhood cardinalities and closeness centrality.
• Introduces a flexible framework that generalizes to a wide class of graph statistics, including those derived from distance distributions.
• Demonstrates that HIP improves estimation quality in HyperLogLog MinHash sketches, outperforming the original estimators in practical data stream applications.
• Employs nearly linear-time computation to construct sketches for all nodes, ensuring scalability on billion-edge graphs.

Experimental results

Research questions

• RQ1 Can a unified, low-variance estimator be developed for All-Distances Sketches to estimate diverse graph statistics accurately?
• RQ2 How does the HIP estimator compare in variance and accuracy to existing estimators for neighborhood cardinality and closeness centrality?
• RQ3 To what extent can HIP improve the performance of approximate distinct counting in data streams, especially in HyperLogLog MinHash sketches?
• RQ4 Is the HIP estimator optimal in terms of variance reduction for distance-related statistics in sketch-based estimation?
• RQ5 Can the HIP framework be applied flexibly and efficiently across a broad class of graph statistics with minimal computational overhead?

Key findings

• The HIP estimator achieves at most half the variance of previous estimators for neighborhood cardinalities and closeness centralities, demonstrating a significant improvement in estimation accuracy.
• HIP estimators are proven to be unbiased and exhibit a polynomial improvement in variance for general statistics, indicating strong theoretical and practical advantages.
• The framework is shown to be essentially optimal in variance reduction for the studied statistics, suggesting minimal room for further improvement under the same sampling model.
• HIP outperforms the original HyperLogLog MinHash estimators in approximate distinct counting, delivering significantly improved estimation quality for data stream applications.
• The method enables scalable, nearly linear-time computation of sketches for all nodes, making it suitable for massive graphs with billions of edges.
• The unified framework supports a wide range of graph statistics, offering a single, elegant solution for diverse distance-based estimations in large-scale networks.