[Paper Review] A Structural Representation Learning for Multi-relational Networks

Xin Li, Huiting Hong+2 more|arXiv (Cornell University)|May 16, 2018

Advanced Graph Neural Networks23 references3 citations

TL;DR

This paper proposes Multi-Relational Network Embedding (MNE), a novel structural representation learning model for multi-relational networks that explicitly preserves triangular and parallelogram connectivity patterns often ignored by existing methods. By replacing the hard constraint $h + r = t$ with a flexible objective function, MNE achieves superior performance on knowledge graph benchmark datasets, outperforming state-of-the-art methods like TransE and LINE, especially in networks with high structural complexity.

ABSTRACT

Most of the existing multi-relational network embedding methods, e.g., TransE, are formulated to preserve pair-wise connectivity structures in the networks. With the observations that significant triangular connectivity structures and parallelogram connectivity structures found in many real multi-relational networks are often ignored and that a hard-constraint commonly adopted by most of the network embedding methods is inaccurate by design, we propose a novel representation learning model for multi-relational networks which can alleviate both fundamental limitations. Scalable learning algorithms are derived using the stochastic gradient descent algorithm and negative sampling. Extensive experiments on real multi-relational network datasets of WordNet and Freebase demonstrate the efficacy of the proposed model when compared with the state-of-the-art embedding methods.

Motivation & Objective

• To address the limitation of existing multi-relational network embedding methods that rely on the hard constraint $h + r = t$, which fails to capture complex structural patterns like triangles and parallelograms.
• To develop a more flexible representation learning framework that implicitly preserves higher-order connectivity structures in multi-relational networks.
• To improve performance on knowledge graph tasks such as triplet classification and link prediction, especially in networks with dense triangular connectivity.
• To enable scalable learning through stochastic gradient descent and negative sampling, supporting large-scale network embedding.

Proposed method

• Proposes a new objective function that implicitly preserves triangular and parallelogram connectivity structures by modeling the geometric relationships between entities and relations without enforcing $h + r = t$.
• Uses a bridging function based on vector addition to model structural patterns, allowing more flexible and accurate embeddings than rigid translation-based models.
• Employs stochastic gradient descent with negative sampling to reduce computational cost and scale to large networks.
• Introduces ${M ext{NE}}^{+}$, an enhanced version of MNE that further improves performance through optimized training dynamics.
• Treats each (head, relation, tail) triplet as a positive sample and generates negative samples by corrupting the head or tail entity.
• Applies a logistic regression classifier on learned embeddings for downstream evaluation in link prediction and triplet classification tasks.

Experimental results

Research questions

• RQ1 Can a multi-relational network embedding model effectively capture triangular connectivity structures that are commonly ignored by existing methods?
• RQ2 Does replacing the hard constraint $h + r = t$ with a more flexible objective function lead to improved representation quality and generalization?
• RQ3 How does the proposed model perform on real-world knowledge graphs with varying levels of structural complexity, such as WordNet and Freebase?
• RQ4 To what extent does the model scale to large networks while maintaining high accuracy?
• RQ5 Can the model achieve better performance than state-of-the-art methods like TransE and LINE in both triplet classification and link prediction tasks?

Key findings

• MNE consistently outperforms all baseline methods, including TransE, TransH, and LINE, on both WN18 and FB15K datasets in triplet classification and link prediction tasks.
• The proposed ${M ext{NE}}^{+}$ model achieves the highest performance across all settings, demonstrating superior generalization and robustness.
• MNE maintains high accuracy even when the proportion of triangular structures increases, while TransE and TransH show significant performance degradation under the same conditions.
• MNE converges to optimal performance at a low embedding dimension (around 20), whereas other methods require at least 100 dimensions to achieve good results, indicating more compact and efficient representations.
• Negative sampling enables scalable training, allowing MNE to handle large-scale networks efficiently without sacrificing performance.