[Paper Review] Estimating Graph Dimension with Cross-validated Eigenvalues

Fan Chen, Sébastien Roch+2 more|arXiv (Cornell University)|Aug 6, 2021

Random Matrices and Applications54 references4 citations

TL;DR

This paper proposes a cross-validation-based method, Eigenvector Cross-Validation (EigCV), to estimate the latent dimension $k$ in random graph models without parametric assumptions. By splitting the adjacency matrix into training and test sets, it computes cross-validated eigenvalues to test whether each sample eigenvector is orthogonal to the true latent space, achieving consistent estimation of $k$ under a broad class of models, especially in sparse or high-dimensional settings where traditional scree plots fail.

ABSTRACT

In applied multivariate statistics, estimating the number of latent dimensions or the number of clusters, $k$, is a fundamental and recurring problem. We study a sequence of statistics called "cross-validated eigenvalues." Under a large class of random graph models, including both Poisson and Bernoulli edges, without parametric assumptions, we provide a $p$-value for each cross-validated eigenvalue. It tests the null hypothesis that the sample eigenvector is orthogonal to (i.e., uncorrelated with) the true latent dimensions. This approach naturally adapts to problems where some dimensions are not statistically detectable. In scenarios where all $k$ dimensions can be estimated, we show that our procedure consistently estimates $k$. In simulations and data example, the proposed estimator compares favorably to alternative approaches in both computational and statistical performance.

Motivation & Objective

• To address the persistent challenge of estimating the number of latent dimensions $k$ in random graph models, especially when scree plots fail due to bias in sample eigenvalues.
• To develop a method that adapts to scenarios where only a subset of the $k$ dimensions are statistically detectable.
• To provide a nonparametric, computationally efficient approach for $k$ estimation that maintains statistical power even in sparse graphs.
• To offer a formal hypothesis test for each eigenvector, assessing its relevance to the true latent structure via cross-validated eigenvalues.

Proposed method

• The method splits the adjacency matrix $A$ into two independent matrices $\tilde{A}$ and $\tilde{A}_{\text{test}}$ via binomial thinning with probability $1-\varepsilon$, ensuring identical distribution under the null.
• It uses the normalized and regularized adjacency matrix $L = DAD$, where $D_{ii} = (d_i + \tau)^{-1/2}$, to stabilize spectral properties in sparse graphs.
• For each eigenvector of $\tilde{A}$, it computes a cross-validated eigenvalue $\lambda_{\text{cv}}$ as the inner product between the eigenvector and the test matrix $\tilde{A}_{\text{test}}$.
• A $p$-value is derived for each eigenvector under the null hypothesis that it is orthogonal to the true latent space, using a central limit theorem for the test statistic.
• The number of significant eigenvectors (those with $p$-values below a threshold) is taken as the estimated $k$, ensuring consistency when all $k$ dimensions are detectable.
• The approach is robust to model misspecification and performs well under both Bernoulli and Poisson edge distributions, with improved performance under higher folds in cross-validation.

Experimental results

Research questions

• RQ1 Can a cross-validation framework improve the detection of latent dimensions in random graphs where traditional scree plots fail due to eigenvalue bias?
• RQ2 How can one formally test whether a sample eigenvector is truly associated with a population eigenvector in the latent space?
• RQ3 Does the proposed method maintain consistency in estimating $k$ when only a subset of the $k$ dimensions are statistically detectable?
• RQ4 How does the method perform in sparse graphs, where standard spectral methods are known to be biased?
• RQ5 What is the impact of the number of cross-validation folds on the Type I error and power of the hypothesis test for each eigenvector?

Key findings

• EigCV consistently estimates $k$ when all $k$ dimensions are detectable, proving consistency under mild identifiability conditions.
• In simulations, EigCV showed lower relative error and reduced variability compared to alternative methods like StGoF, BHMC, and NCV, especially in sparse graphs.
• On a real email network with 22,688 nodes, EigCV estimated $k = 30.56$, significantly closer to the true dimension than BHMC ($14.00$) and LR ($13.00$), with faster runtime (0.81s vs. 128.17s).
• The method remains robust under both Bernoulli and Poisson edge models, with the Poisson model yielding more accurate standard error estimates than the Bernoulli model.
• Increasing the number of folds in cross-validation reduces the variance of test statistics, making $T_2$ more powerful but $T_3$ and $T_4$ more conservative, though their null distributions remain approximately normal.
• The cross-validated eigenvalue test provides a principled, nonparametric way to assess the statistical significance of each eigenvector, avoiding arbitrary elbow detection in scree plots.