[Paper Review] Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2vec

Jiezhong Qiu, Yuxiao Dong+4 more|arXiv (Cornell University)|Oct 9, 2017

Advanced Graph Neural Networks45 references69 citations

TL;DR

The paper shows that DeepWalk, LINE, PTE, and node2vec are implicit matrix factorizations of graph-derived matrices and introduces NetMF to explicitly factorize a DeepWalk/Laplacian-based matrix, improving embedding performance.

ABSTRACT

Since the invention of word2vec, the skip-gram model has significantly advanced the research of network embedding, such as the recent emergence of the DeepWalk, LINE, PTE, and node2vec approaches. In this work, we show that all of the aforementioned models with negative sampling can be unified into the matrix factorization framework with closed forms. Our analysis and proofs reveal that: (1) DeepWalk empirically produces a low-rank transformation of a network's normalized Laplacian matrix; (2) LINE, in theory, is a special case of DeepWalk when the size of vertices' context is set to one; (3) As an extension of LINE, PTE can be viewed as the joint factorization of multiple networks' Laplacians; (4) node2vec is factorizing a matrix related to the stationary distribution and transition probability tensor of a 2nd-order random walk. We further provide the theoretical connections between skip-gram based network embedding algorithms and the theory of graph Laplacian. Finally, we present the NetMF method as well as its approximation algorithm for computing network embedding. Our method offers significant improvements over DeepWalk and LINE for conventional network mining tasks. This work lays the theoretical foundation for skip-gram based network embedding methods, leading to a better understanding of latent network representation learning.

Motivation & Objective

• Clarify theoretical connections between popular skip-gram based network embedding methods and graph Laplacians.
• Derive closed-form matrices that each method implicitly factorizes.
• Propose NetMF to explicitly factorize the DeepWalk/Laplacian-inspired matrix and assess its performance.
• Demonstrate empirical gains of NetMF over DeepWalk and LINE on standard datasets.

Proposed method

• Prove that DeepWalk, LINE, PTE, and node2vec correspond to implicit matrix factorizations of specific network-derived matrices.
• Show LINE as a special case of DeepWalk with window size T = 1.
• Extend LINE analysis to PTE as joint factorization of multiple sub-networks.
• Model node2vec with a 2nd-order random walk and derive its matrix-like factorization form.
• Introduce NetMF to explicitly factorize the DeepWalk matrix (or its log) via SVD, with two practical schemes for small and large window sizes.

Experimental results

Research questions

• RQ1 What is the precise matrix form underlying each of DeepWalk, LINE, PTE, and node2vec?
• RQ2 How are these methods theoretically connected to the graph Laplacian and spectral properties of networks?
• RQ3 Can we design an explicit matrix factorization method (NetMF) that matches or surpasses these methods in practice?
• RQ4 What are the empirical benefits of explicit factorization (NetMF) over implicit sampling-based methods on real networks?

Key findings

• All four methods are equivalent to implicit matrix factorizations of closed-form matrices.
• LINE (2nd) is a special case of DeepWalk when the context window T = 1.
• PTE extends LINE as a joint factorization across multiple sub-networks.
• For node2vec, a 2nd-order walk yields a factorization related to a transition tensor, though full matrix form is complex.
• NetMF explicitly factorizes the DeepWalk/Laplacian-inspired matrix, with small-T and large-T variants.
• NetMF (T=1 and T=10) shows substantial gains over LINE and DeepWalk on multiple datasets (e.g., BlogCatalog, PPI, Wikipedia, Flickr).