[Paper Review] Bias-Variance Tradeoffs in Joint Spectral Embeddings

Benjamin Draves, Daniel L. Sussman|arXiv (Cornell University)|May 5, 2020

Complex Network Analysis Techniques47 references4 citations

TL;DR

This paper analyzes the omnibus embedding for joint spectral embeddings in heterogeneous network data, establishing an explicit finite-sample bias-variance tradeoff. It derives analytical expressions for bias, concentration bounds, and asymptotic normality of latent position estimates, enabling valid inference despite estimator inconsistency, and demonstrates improved performance in community detection and hypothesis testing under the Eigen-Scaling Random Dot Product Graph model.

ABSTRACT

Joint spectral embeddings facilitate analysis of multiple network data by simultaneously mapping vertices in each network to points in Euclidean space where statistical inference is then performed. In this work, we consider one such joint embedding technique, the omnibus embedding of arXiv:1705.09355 , which has been successfully used for community detection, anomaly detection, and hypothesis testing tasks. To date the theoretical properties of this method have only been established under the strong assumption that the networks are conditionally i.i.d. random dot product graphs. Herein, we take a first step in characterizing the theoretical properties of the omnibus embedding in the presence of heterogeneous network data. Under a latent position model, we show the omnibus embedding implicitly regularizes its latent position estimates which induces a finite-sample bias-variance tradeoff for latent position estimation. We establish an explicit bias expression, derive a uniform concentration bound on the residual, and prove a central limit theorem characterizing the distributional properties of these estimates. These explicit bias and variance expressions enable us to state sufficient conditions for exact recovery in community detection tasks and develop a pivotal test statistic to determine whether two graphs share the same set of latent positions; demonstrating that accurate inference is achievable despite the estimator's inconsistency. These results are demonstrated in several experimental settings where statistical procedures utilizing the omnibus embedding are competitive, and oftentimes preferable, to comparable embedding techniques. These observations accentuate the viability of the omnibus embedding for multiple graph inference beyond the homogeneous network setting.

Motivation & Objective

• To characterize the finite-sample behavior of the omnibus embedding under heterogeneous network models, extending beyond the i.i.d. assumption.
• To identify and quantify the implicit bias-variance tradeoff induced by the omnibus embedding in latent position estimation.
• To establish theoretical guarantees—bias expression, concentration, and asymptotic normality—for latent position estimates in the presence of network heterogeneity.
• To enable valid statistical inference, including exact recovery in community detection and pivotal hypothesis testing, despite estimator inconsistency.

Proposed method

• Proposes the Eigen-Scaling Random Dot Product Graph (ESRDPG) as a heterogeneous network model that extends the RDPG to multiplex networks.
• Derives an explicit analytical expression for the finite-sample bias of the omnibus embedding's latent position estimates under the ESRDPG.
• Establishes a uniform concentration bound on the residual error of the latent position estimates.
• Proves a central limit theorem for the latent position estimates, showing their asymptotic normality with a known covariance structure.
• Develops a pivotal test statistic based on the asymptotic distribution to test whether two graphs share the same latent positions.
• Uses the second-order delta method and Slutsky’s theorem to derive the asymptotic distribution of test statistics under both null and alternative hypotheses.

Experimental results

Research questions

• RQ1 What is the nature of the finite-sample bias in the omnibus embedding when applied to heterogeneous network data?
• RQ2 How does the omnibus embedding's implicit regularization induce a bias-variance tradeoff in latent position estimation?
• RQ3 Can valid statistical inference be performed despite the inconsistency of the omnibus embedding estimator under heterogeneous models?
• RQ4 What are the sufficient conditions for exact recovery in community detection using the omnibus embedding under heterogeneous networks?
• RQ5 Can a pivotal test statistic be constructed to determine whether two graphs share the same latent positions, even when the estimator is inconsistent?

Key findings

• The omnibus embedding induces a finite-sample bias in latent position estimates under the ESRDPG model, with an explicit analytical expression derived.
• A uniform concentration bound is established for the residual error in the latent position estimates, ensuring control over estimation variability.
• The asymptotic distribution of the latent position estimates is shown to be a mixture of normal distributions with a known covariance matrix, enabling rigorous inference.
• The pivotal test statistic $ W_i $ asymptotically follows a $ \chi^2_d $ distribution under the null hypothesis, allowing for valid hypothesis testing.
• The test statistic maintains power under the alternative, with the asymptotic distribution depending on the difference in graph-specific transformation matrices $ \mathbf{S}^{(1)} $ and $ \mathbf{S}^{(2)} $.
• Despite the estimator's inconsistency, the theoretical framework enables exact recovery in community detection and competitive performance in experimental settings.