[Paper Review] Multi-Level Network Embedding with Boosted Low-Rank Matrix Approximation

Jundong Li, Liang Wu+1 more|arXiv (Cornell University)|Aug 26, 2018

Advanced Graph Neural Networks35 references3 citations

TL;DR

This paper proposes BoostNE, a multi-level network embedding framework that learns node representations in a stagewise, boosting-like manner to overcome the limitations of global low-rank matrix approximation in traditional network embedding. By iteratively refining residual connectivity matrices through low-rank approximation, BoostNE captures hierarchical connectivity patterns across multiple granularities, achieving state-of-the-art performance on node classification and link prediction tasks.

ABSTRACT

As opposed to manual feature engineering which is tedious and difficult to scale, network representation learning has attracted a surge of research interests as it automates the process of feature learning on graphs. The learned low-dimensional node vector representation is generalizable and eases the knowledge discovery process on graphs by enabling various off-the-shelf machine learning tools to be directly applied. Recent research has shown that the past decade of network embedding approaches either explicitly factorize a carefully designed matrix to obtain the low-dimensional node vector representation or are closely related to implicit matrix factorization, with the fundamental assumption that the factorized node connectivity matrix is low-rank. Nonetheless, the global low-rank assumption does not necessarily hold especially when the factorized matrix encodes complex node interactions, and the resultant single low-rank embedding matrix is insufficient to capture all the observed connectivity patterns. In this regard, we propose a novel multi-level network embedding framework BoostNE, which can learn multiple network embedding representations of different granularity from coarse to fine without imposing the prevalent global low-rank assumption. The proposed BoostNE method is also in line with the successful gradient boosting method in ensemble learning as multiple weak embeddings lead to a stronger and more effective one. We assess the effectiveness of the proposed BoostNE framework by comparing it with existing state-of-the-art network embedding methods on various datasets, and the experimental results corroborate the superiority of the proposed BoostNE network embedding framework.

Motivation & Objective

• Address the fundamental limitation of global low-rank matrix approximation in network embedding, which fails to capture complex, non-low-rank connectivity patterns in real-world networks.
• Overcome the inadequacy of single-factorization approaches that cannot model diverse node interaction patterns across different granularities.
• Develop a multi-level framework that learns node representations from coarse to fine granularity, mimicking the success of gradient boosting in ensemble learning.
• Relax the global low-rank assumption by decomposing residual matrices at each stage, enabling progressive refinement of embedding representations.
• Provide a unified, scalable framework compatible with matrix factorization-based network embedding while improving expressiveness and generalization.

Proposed method

• Formulate the network embedding problem within a matrix factorization framework, using a closed-form connectivity matrix derived from node proximity or structural patterns.
• Apply low-rank matrix approximation (e.g., SVD, NMF) to the connectivity matrix to generate an initial embedding representation.
• Compute the residual matrix as the difference between the original connectivity matrix and its low-rank approximation to capture unexplained connectivity patterns.
• Iteratively apply low-rank approximation to the residual matrix in a stagewise fashion, generating a sequence of increasingly refined embedding representations.
• Combine the multiple weak embedding representations from each stage into a final, stronger, and more discriminative embedding via boosting-like aggregation.
• Ensure computational scalability by leveraging efficient low-rank factorization techniques and avoiding full matrix reconstruction at each step.

Experimental results

Research questions

• RQ1 Can a multi-level embedding framework that iteratively refines residual connectivity matrices outperform single-level low-rank matrix factorization in capturing complex network structures?
• RQ2 Does the boosting-inspired stagewise learning of multiple low-rank approximations improve node representation quality compared to state-of-the-art network embedding methods?
• RQ3 To what extent does relaxing the global low-rank assumption enhance the expressiveness of learned node embeddings across diverse network types?
• RQ4 How do different levels of granularity in the embedding hierarchy contribute to downstream tasks like node classification and link prediction?
• RQ5 Can the proposed framework maintain computational efficiency while improving representation quality on large-scale networks?

Key findings

• BoostNE achieves state-of-the-art performance on node classification and link prediction tasks across multiple real-world datasets, outperforming existing SOTA methods.
• The multi-level framework significantly improves embedding quality by capturing hierarchical and non-global low-rank patterns that single-factorization methods miss.
• The ablation study confirms that each stage of the boosting process contributes meaningfully to the final representation, with performance improving monotonically across levels.
• The framework demonstrates robustness across diverse network types, including social, citation, and collaboration networks, indicating broad applicability.
• The residual matrix decomposition strategy effectively isolates and models complex connectivity patterns not captured by the initial low-rank approximation.
• The method maintains computational feasibility on large-scale networks by relying on efficient low-rank factorization and incremental updates.