[Paper Review] Sampling-based Estimation of In-degree Distribution with Applications to Directed Complex Networks

Nelson Antunes, Shankar Bhamidi+3 more|arXiv (Cornell University)|Oct 2, 2018

Complex Network Analysis Techniques33 references4 citations

TL;DR

This paper proposes a sampling-based method to estimate the in-degree distribution in large directed networks when only out-edges are observable. It combines an inversion approach for the bulk distribution and an asymptotic method for the tail, achieving accurate recovery of the full in-degree distribution at sampling rates of 15–20% across synthetic and real networks, with superior performance for power-law tails using random walk sampling with 30% jump rates.

ABSTRACT

The focus of this work is on estimation of the in-degree distribution in directed networks from sampling network nodes or edges. A number of sampling schemes are considered, including random sampling with and without replacement, and several approaches based on random walks with possible jumps. When sampling nodes, it is assumed that only the out-edges of that node are visible, that is, the in-degree of that node is not observed. The suggested estimation of the in-degree distribution is based on two approaches. The inversion approach exploits the relation between the original and sample in-degree distributions, and can estimate the bulk of the in-degree distribution, but not the tail of the distribution. The tail of the in-degree distribution is estimated through an asymptotic approach, which itself has two versions: one assuming a power-law tail and the other for a tail of general form. The two estimation approaches are examined on synthetic and real networks, with good performance results, especially striking for the asymptotic approach.

Motivation & Objective

• To estimate the in-degree distribution in large directed networks when only out-edges are visible during sampling.
• To address the challenge of ill-conditioned inverse problems arising from sampling with limited visibility of in-degrees.
• To develop a robust estimation framework that captures both the bulk and tail of the in-degree distribution.
• To evaluate performance across diverse network topologies, including power-law and non-power-law structures.
• To provide a scalable, computationally efficient method applicable to real-world networks such as citation, web, and financial networks.

Proposed method

• Uses random vertex sampling (RVS) and random edge sampling (RES) as baseline schemes to model sampling processes.
• Models the estimation as a linear inverse problem involving a sampling matrix that maps true in-degree distribution to observed sample distribution.
• Applies penalized weighted least-squares to mitigate ill-conditioning in the inverse problem for low sampling rates.
• Employs an asymptotic approach based on probabilistic equivalence between true and sampled tail distributions.
• Develops two asymptotic variants: LINE (assuming power-law tail) and ASYM (distribution-free tail estimation).
• Uses random walks with jumps to simulate realistic sampling schemes that converge to uniform sampling in the limit.

Experimental results

Research questions

• RQ1 How can the in-degree distribution be reliably estimated when only out-edges are observable in sampled nodes?
• RQ2 What sampling schemes (e.g., random walk with jumps) best preserve the structure of the true in-degree distribution?
• RQ3 How does the performance of inversion-based estimation compare to asymptotic tail estimation across different network types?
• RQ4 To what extent can the full in-degree distribution be recovered at low sampling rates (15–20%)?
• RQ5 How do jump rates in random walks affect tail estimation accuracy, especially for power-law networks?

Key findings

• The inversion approach accurately recovers the bulk of the in-degree distribution under sampling rates of 15–20%.
• The asymptotic approach, particularly the LINE method assuming power-law tails, achieves strong performance on networks with heavy-tailed in-degree distributions.
• For power-law networks, random walk sampling with 30% jump rates yields optimal tail estimation performance.
• The inversion method is robust to variations in random walk sampling schemes and jump rates.
• The combined inversion and asymptotic approach successfully recovers the full in-degree distribution across synthetic and real-world networks, including Amazon co-purchasing and citation networks.
• The method remains computationally efficient and effective even when in-degrees are not directly observable during sampling.