[Paper Review] Fast and Accurate Network Embeddings via Very Sparse Random Projection

Haochen Chen, Syed Fahad Sultan+3 more|arXiv (Cornell University)|Aug 30, 2019

Advanced Graph Neural Networks35 references4 citations

TL;DR

FastRP is a highly scalable network embedding method that constructs a normalized node similarity matrix capturing high-order proximities and applies very sparse random projection for dimensionality reduction. It achieves over 4,000x speedup over DeepWalk while matching or exceeding its performance on downstream tasks like node classification and link prediction.

ABSTRACT

We present FastRP, a scalable and performant algorithm for learning distributed node representations in a graph. FastRP is over 4,000 times faster than state-of-the-art methods such as DeepWalk and node2vec, while achieving comparable or even better performance as evaluated on several real-world networks on various downstream tasks. We observe that most network embedding methods consist of two components: construct a node similarity matrix and then apply dimension reduction techniques to this matrix. We show that the success of these methods should be attributed to the proper construction of this similarity matrix, rather than the dimension reduction method employed. FastRP is proposed as a scalable algorithm for network embeddings. Two key features of FastRP are: 1) it explicitly constructs a node similarity matrix that captures transitive relationships in a graph and normalizes matrix entries based on node degrees; 2) it utilizes very sparse random projection, which is a scalable optimization-free method for dimension reduction. An extra benefit from combining these two design choices is that it allows the iterative computation of node embeddings so that the similarity matrix need not be explicitly constructed, which further speeds up FastRP. FastRP is also advantageous for its ease of implementation, parallelization and hyperparameter tuning. The source code is available at https://github.com/GTmac/FastRP.

Motivation & Objective

• Address the scalability bottleneck in state-of-the-art network embedding methods, which suffer from high constant factors due to sampling and expensive optimization in dimension reduction.
• Identify that the quality of network embeddings stems primarily from proper similarity matrix construction rather than the choice of dimension reduction technique.
• Develop a method that decouples similarity matrix construction from expensive optimization, enabling iterative, memory-efficient computation.
• Achieve both high performance and extreme speedup on large-scale graphs without sacrificing representation quality.

Proposed method

• Construct a high-order proximity-based node similarity matrix using powers of the adjacency matrix, capturing transitive relationships across multiple hops.
• Apply a novel normalization scheme that downweights high-degree nodes by leveraging the convergence properties of $\mathbf{A}^k$, formalized as a matrix multiplication for efficiency.
• Utilize very sparse random projection (VSRP) as a scalable, optimization-free dimension reduction technique, replacing computationally heavy models like Skip-gram.
• Enable iterative computation of embeddings by combining normalization and VSRP, avoiding explicit storage of the full similarity matrix.
• Formalize the entire pipeline as a sequence of matrix operations to support parallelization and efficient implementation.
• Leverage the mathematical structure of the normalized similarity matrix and VSRP to allow streaming or incremental computation, reducing memory overhead.

Experimental results

Research questions

• RQ1 Can a network embedding method achieve both high speed and high accuracy by decoupling similarity matrix construction from expensive optimization?
• RQ2 What is the impact of proper node degree normalization on the quality of learned embeddings in sparse, real-world graphs?
• RQ3 How does very sparse random projection compare to traditional dense or Gaussian random projections in terms of scalability and representation quality?
• RQ4 Can iterative computation of embeddings be enabled through the combination of normalized similarity matrices and sparse random projection?
• RQ5 To what extent can the performance of state-of-the-art methods like DeepWalk be surpassed in speed while maintaining or improving accuracy?

Key findings

• FastRP achieves over 4,000x speedup compared to DeepWalk on the YouTube graph, completing embedding in minutes rather than days.
• FastRP produces embeddings of comparable or better quality than DeepWalk and node2vec on node classification and link prediction tasks across multiple real-world networks.
• The normalized similarity matrix construction significantly improves embedding quality by mitigating the dominance of high-degree nodes, a factor overlooked in prior work.
• Very sparse random projection enables high-speed dimensionality reduction without requiring iterative optimization, unlike Skip-gram-based methods.
• The iterative computation framework allows FastRP to avoid explicitly materializing the full similarity matrix, reducing memory usage and enabling scalability.
• The method is easily parallelizable and requires minimal hyperparameter tuning, enhancing practical deployment ease.